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Abstract 

The  source  selection  process  for  choosing  a  contractor  does  not  incorporate  a 
standardized  objective  decision  analysis  tool;  therefore,  the  process  is  extremely 
subjective  and  provides  little  guidance  to  distinguish  between  highly  competitive 
contractors.  The  Air  Force  Simplified  Acquisition  of  Base  Engineer  Requirements 
(SABER)  program  selects  contractors  through  a  Low  Price  Technically  Acceptable 
(LPT  A)  source  selection  process  and  encounters  the  same  problem  of  not  being  able  to 
objectively  distinguish  between  the  competing  contractors.  The  LPTA  process  rank 
orders  the  contractors  based  on  price  and  evaluates  the  bidders  in  order  until  an 
“exceptional”  contractor  is  discovered.  However,  the  SABER  source  selection 
committee  members  wish  to  evaluate  all  contractors  using  all  decision  criteria  with  the 
ability  to  objectively  compare  all  contractors  to  one  another. 

Since  there  are  several  factors  and  guidelines  to  consider  when  awarding  a 
SABER  contract,  a  value  focused  thinking  approach  was  used  to  create  a  structured 
decision  making  model  that  takes  into  account  all  values  along  with  their  desired 
weighting  as  specified  by  members  of  a  SABER  source  selection  team.  The  model  was 
then  used  to  evaluate  seven  contractors  who  recently  competed  for  a  SABER  contract  and 
perform  deterministic  and  sensitivity  analysis  on  the  recommended  decision  outcome. 

The  results  of  this  research  illustrate  the  valuable  insight  and  practicality  of  applying  a 
quantitative,  objective,  consistent,  and  defendable  tool  for  SABER  source  selections. 

The  value  gained  from  this  model  will  potentially  aid  the  SABER  source  selection 
process,  as  well  as  other  government  and  private/public  source  selections. 
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A  DECISION  ANALYSIS  TOOL  LOR  THE 


SOURCE  SELECTION  PROCESS 


Chapter  I.  Introduction 

The  source  selection  process  is  an  extremely  detailed  and  time  consuming  course  of 
action  utilized  to  select  the  best  offeror  responding  to  a  specific  solicitation.  This 
process,  applied  by  both  private  and  public  sector  organizations,  consists  of  several 
regulations  and  evaluating  processes.  However,  it  does  not  incorporate  any  techniques 
for  objective  decision  analysis.  To  focus  the  application  of  this  thesis,  an  in-depth 
research  effort  on  the  process  of  an  Air  Force  Simplified  Acquisition  of  Base  Engineer 
Requirements  (SABER)  contractor  source  selection  will  be  accomplished.  Although  an 
Air  Force  SABER  source  selection  will  be  the  subject  focus,  the  problems  addressed  and 
methodology  applied  seem  relevant  to  all  private  and  public  sector  source  selection 
processes. 

1.1  Background 

Air  Force  guidance  states  that  a  SABER  contract  provides  a  streamlined  means  to 
complete  construction  projects  estimated  at  less  than  $750,000.  In  fact,  the  guidance 
states  that  its  main  purpose  is  to  expedite  the  award  of  civil  engineer  (CE)  requirements 
for  projects  typically  ranging  from  $50,000  to  $500,000.  As  such,  a  SABER  contract 
means  a  fixed-price,  indefinite-delivery/indefinite-quantity  (ID/IQ)  contract 
(IG5336.9201,  2005).  Typical  ID/IQ  construction  contracts  are  for  single  trades, 
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meaning  they  only  operate  in  the  realm  of  one  type  of  construction  (i.e.,  pavement, 
roofing,  or  painting).  ID/IQ  contracts  are  awarded  with  the  understanding  that  a  large 
quantity  of  separate  projects  will  be  accomplished  over  a  period  of  time  (usually  over  4  to 
5  years);  however,  it  is  not  certain  when  each  individual  project  will  occur.  Therefore, 
the  contract  is  issued  using  a  general  specification  guide,  pricing  guide,  and  Request  for 
Proposal  (RFP)  to  fully  encompass  several  years  of  potential  construction  work. 

A  SABER  contract  makes  it  quick  and  efficient  to  issue  projects  throughout  the  tenn 
of  the  contract.  As  with  most  ID/IQ  contracts,  a  SABER  contract  usually  consists  of  a 
“base  year  with  four  optional  years  for  a  total  contract  duration  of  five  years”  (Henry  and 
Brothers,  2001).  However,  unlike  a  typical  ID/IQ  contract,  SABER  is  unique  because  it 
is  a  multi-trade  contract  and  is  intended  for  small  scale  construction.  Therefore,  this 
contracting  mechanism  is  best  suited  for  non-complex,  minor  construction,  and 
maintenance  and  repair  projects  that  require  minimum  design;  it  is  not  an  appropriate  for 
Architect-Engineering  (A-E)  services  (IG5336.9201,  2005). 

The  main  components  of  a  SABER  contract  include  detailed  specifications,  Unit 
Pricing  Guides  (UPG),  and  coefficients.  The  specifications  used  in  a  SABER  contract 
are  prepared  by  the  base  civil  engineer  (BCE)  and  include  the  master  specifications, 
which  describe  the  overall  scope  of  the  contract,  and  the  technical  specifications,  which 
define  specific  construction  standards  for  tasks  ordered  under  the  contract.  After 
developing  the  specifications,  the  BCE  selects  a  Unit  Price  Guide  (UPG).  UPGs  are 
commercial  pricing  tools,  such  as  computer  cost  databases  and  libraries  of  hard  copy 
books,  that  list  tasks  by  unit  of  measure  and  unit  price  (IG5336.9201,  2005).  Examples 
of  commercially  available  UPGs  are  WinEstimator  Inc.,  Timberline  Software,  and  R.S. 
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MEANS,  Inc.  (IG5336.9201,  2005).  Typically,  SABER  contracts  utilize  R.S.  MEANS. 
Prices  are  general  in  that  differing  pricing  rates  for  different  parts  of  the  country  are  not 
accounted  for;  therefore,  UPGs  must  undergo  localization  -  tailoring  the  prices  to  the 
specific  city  or  general  location  in  which  the  UPG  will  be  utilized.  The  final  element  in 
the  pricing  of  a  SABER  contract  is  the  coefficients.  Coefficients  are  factors  multiplied 
against  the  localized  unit  prices  in  the  UPG  to  calculate  the  finalized  individual  task  order 
(TO)  price  (IG5336.9201,  2005).  These  coefficients,  proposed  by  contractors  in  their 
bidding  documents,  are  intended  to  represent  the  contractor’s  costs  for  overhead,  profit, 
minimum  design  costs,  general  and  administrative  expenses,  bond  premiums,  and  gross 
receipt  taxes  (IG5336.9201,  2005). 

Two  main  advantages  gained  by  implementing  a  successful  SABER  program 
include:  1)  Improved  customer  service  and  responsiveness  and  2)  Incentives  for  the 
contractor  to  work  to  a  high  standard,  and  complete  projects  in  a  timely  manner  in  order 
to  receive  TOs  for  future  projects  (IG5336.9201,  2005).  Other  advantages  include 
(AFP AM  32-1005,  1999) 

1)  Enhanced  ability  to  accomplish  backlogged  work  orders  and  commander¬ 
generated  requirements  -  an  easy  way  to  quickly  accommodate  hot  projects; 

2)  Potential  for  greatly  improved  working  relationships  and  synergy  between 
Base  Civil  Engineer  (BCE),  contracting,  and  the  contractor; 

3)  Addition  of  resources  to  the  BCE  -  a  SABER  contractor  is  unaffected  by 
deployments,  training,  or  inspections;  and 
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4)  Added  fiscal  flexibility  -  other  units  are  more  willing  to  fund  project  that  are 
quickly  responded  to  and  completed  in  a  timely  manner  with  high  quality 
performance. 

The  selection  of  a  SABER  contractor  is  conducted  using  an  Air  Force  Source 
Selection  process.  A  source  selection  committee,  usually  consisting  of  a  Contracting 
Officer  (CO)  and  CE  personnel,  is  fonned  to  evaluate  and  award  the  contract  “based  on 
the  contractor  that  has  the  best  capability,  capacity,  and  coefficient  mix,  as  detennined  by 
the  contractor  selection  criteria”  (Henry  and  Brothers,  2001).  The  Source  Selection 
Authority  (SSA),  often  the  CO,  will  make  the  final  award  decision  and  has  wide  latitude 
and  discretion  in  how  to  run  a  source  selection.  The  contractor  selection  criteria  are 
specified  in  the  RFP. 

The  contractor’s  subsequent  proposal  will  usually  consist  of  two  volumes  -  a  cost 
volume  and  a  technical  volume.  The  cost  volume  will  be  evaluated  by  the  cost  analyst, 
usually  the  CO,  and  the  technical  volume  will  be  evaluated  by  the  technical  analysts,  the 
CE  personnel  committed  to  the  source  selection  team.  The  justification  for  conducting 
the  evaluation  in  two  separate  entities  is  so  the  technical  analysts  are  not  unduly 
influenced  by  the  costs  presented  by  the  contractors. 

1.2  Problem  Background 

Although  mandatory  procedures  exist  governing  the  overall  source  selection  process, 
SABER  contractors  are  usually  selected  under  the  Lowest  Price  Technically  Acceptable 
(LPT A)  Source  Selection  Process.  An  LPTA  source  selection  rank  orders  the  contractors 
by  price  (lowest  to  highest).  Beginning  with  the  contractor  with  the  lowest  price,  their 
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past  performance  is  evaluated.  The  first  contractor  with  an  exceptional  past  performance 
rating  is  awarded  the  contract.  Under  this  selection  technique,  only  the  lowest  bidding 
contractors  are  evaluated;  in  fact,  it  is  possible  that  only  one  contractor  is  actually 
evaluated. 

According  to  Federal  Acquisition  Regulation  (FAR)  15.101-2  (b)(1),  solicitations 
shall  specify  that  award  will  be  made  on  the  basis  of  the  lowest  evaluated  price  of 
proposals  meeting  or  exceeding  the  acceptability  standards  for  non-cost  factors. 
Furthermore,  if  past  performance  is  to  be  included  as  an  evaluation  factor  (as  is  the  case 
for  SABER  source  selections),  it  shall  be  evaluated  in  accordance  with  FAR  15.305. 

This  is  critical  since  FAR  15.305  (2005)  defines  proposal  evaluation  as  “an  assessment  of 
the  proposal  and  the  offeror’s  ability  to  perform  the  prospective  contract  successfully.” 
Under  this  guidance,  an  agency  is  required  to  evaluate  competitive  proposals  and  assess 
their  relative  qualities  based  solely  on  the  factors  and  subfactors  specified  in  the 
solicitation.  A  source  selection  committee  may  get  the  impression  that  they  will  be  able 
to  evaluate  all  the  offerors  according  to  this  guidance;  however,  that  is  not  how  an  LPTA 
source  selection  is  conducted  as  previously  discussed. 

The  format  for  SABER  RFPs  is  standard  across  the  Air  Force  except  for  base- 
specific  information.  The  SABER  RFP  states,  “For  those  proposals  determined  to  be 
acceptable,  a  tradeoff  between  price  and  past  performance  will  be  conducted  with  past 
performance  being  significantly  more  important  than  price”  (Section  M,  Paragraph  2.0). 
From  this  information,  if  past  perfonnance  and  price  were  weighted  factors  and  all 
possible  weighting  scenarios  were  considered,  past  performance  would  assume  at  least 
51%  of  the  decision  since  it  is  supposed  to  be  significantly  more  important  than  price. 
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The  significance  of  this  infonnation  will  become  more  evident  within  the  following 
chapters  of  this  thesis.  The  SABER  RFP  also  states,  “If  the  lowest  priced  acceptable 
offer  received  a  perfonnance  confidence  assessment  rating  of  ‘exceptional/high 
confidence’,  that  offer  represents  the  best  value  for  the  Government.  Award  shall  be 
made  to  that  offeror  without  further  consideration  of  any  other  offerors”  (Section  M, 
Subsection  B,  Paragraph  B).  This  reiterates  the  process  of  an  LPTA  source  selection  and 
describes  how  only  the  lowest  priced  contractors  will  be  evaluated. 

The  intent  of  the  government  is  to  select  a  SABER  contractor  that  represents  the  best 
value;  however,  the  current  source  selection  process  focuses  on  the  lowest  bidding 
contractors.  The  selection  committee  does  not  get  to  evaluate  all  the  offerors,  and  the 
pricing  influence  on  the  decision  is  supposed  to  be  outweighed  by  the  past  perfonnance 
influence.  However,  there  is  no  way  of  ensuring  a  weighting  value  of  5 1%  or  greater  for 
past  performance  will  hold  constant  throughout  the  source  selection.  Additionally,  the 
technical  evaluations  are  obtained  from  past  perfonnance  reviews  and  simply  recorded  on 
a  single  evaluation  sheet  per  contractor  (overall  process  explained  in  Chapter  2).  This 
method  of  consolidating  all  factors  considered  in  detennining  a  qualified  SABER 
contractor  does  not  allow  for  proper  identification  and  analysis  of  how  each  factor 
potentially  influences  the  ratings  of  each  contractor.  The  technical  analysts  are  not  able 
to  properly  compare  contractors  to  one  another  based  on  the  factors  used  for  the 
evaluation,  nor  are  they  able  to  objectively  differentiate  between  the  ratings  assigned  to 
the  contractors  utilizing  the  current  source  selection  method. 
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1.3  Problem  Statement 


When  competing  contractors  have  similar  qualities  based  on  price  and  past 
performance,  it  is  difficult  to  detennine  which  contractor  represents  the  true  best  value  to 
the  Air  Force.  In  many  source  selection  cases,  multiple  competing  contractors  are  highly 
qualified,  have  impeccable  past  performance  reviews,  and  meet  pricing  expectations.  In 
a  LPTA  Source  Selection,  the  lowest  bidder  (with  an  exceptional  rating)  will  win  the 
contract  award  but  may  not  be  the  best  contractor.  Evidence  that  low-priced,  technically 
acceptable  contractors  may  not  be  in  the  best  interests  of  the  Air  Force  was  noted  by 
Heaps  (2001): 

Recently,  the  Assistant  Secretary  of  the  Air  Force  for  Acquisition’s  Operational 
Contracting  Division  (SAF/AQCO)  identified  a  number  of  installations  that  are 
having  problems  with  its  SABER  contractors.  Some  contractors  have  gone  out 
of  business  and  defaulted  on  their  contracts.  Other  contractors  have  not  lived  up 
to  the  performance  standards  identified  in  the  SABER  contract.  In  these 
situations,  the  Government  has  decided  not  to  exercise  the  next  option  of  the 
existing  SABER  contract. 

Unsuccessful  SABER  contractors  are  an  immense  waste  of  time  and  money.  To  select  a 
successful  contractor,  source  selection  teams  need  to  have  a  firm  understanding  of  the 
attributes  and  criteria,  beyond  price  and  past  perfonnance,  impacting  their  decision. 
Furthermore,  decision  makers  must  have  the  latitude  to  be  able  to  analyze  how  the 
contractors  compare  against  each  another. 

The  decision  process  for  source  selection  teams  typically  does  not  include  a 
definitive  proposal  evaluation  tool  capable  of  converting  qualitative  judgmental  criteria 
beyond  price  and  past  performance  into  measurable  quantitative  data  which  can  be  easily 
interpreted.  There  is  a  need  for  a  tool  that  can  provide  objective,  accurate,  adaptable, 
defendable,  and  quantifiable  analysis.  Also,  the  tool  needs  to  have  the  capability  to  easily 
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delineate  between  competing  contractors.  Therefore,  the  motivating  factor  of  this  thesis 
is  to  develop  a  decision  analysis  tool  that  can  be  incorporated  into  the  SABER  source 
selection  process  to  illustrate  the  importance  of  fully  identifying  all  criteria  and 
evaluating  all  the  alternatives  through  a  systematic  and  objective  model. 

1.4  Research  Question 

Because  SABER  source  selections  are  primarily  based  on  price  and  past 
performance,  other  criteria  and  underlying  measures  vital  to  the  decision  process  may  be 
neglected  or  not  fully  understood  by  the  decision  makers.  When  a  difficult  contractor 
selection  is  made,  the  decision  makers  may  not  be  comfortable  or  confident  when  having 
to  explain  and  document  the  final  outcome,  especially  when  all  the  offerors  may  not  have 
been  evaluated.  Therefore,  this  research  is  focused  on  the  following  multi-faceted 
question:  What  are  all  the  criteria,  values,  and  measures  determined  essential  by  the 
decision  makers  to  effectively  select  the  best  contractor  and  why  is  it  essential  to  evaluate 
all  the  alternatives? 

1.5  Investigative  Questions 

To  help  facilitate  an  understanding  of  how  other  criteria  and  underlying  measures 
can  influence  a  decision  maker,  as  well  as  adequately  answer  the  research  question,  the 
following  investigative  questions  will  need  to  be  answered: 

•  How  well  does  the  current  selection  process  work? 
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•  What  are  the  criteria,  to  include  price  and  past  performance,  that  are  deemed 
necessary  to  evaluate  competing  contractors  and  what  are  the  underlying 
measures  for  these  criteria? 

•  How  are  the  measures  weighted  by  the  decision  makers? 

•  How  well  does  the  model  convey  the  true  feelings  of  the  decision  makers? 

•  How  does  the  model  compare  to  the  current  selection  process? 

For  the  purpose  of  this  research,  several  iterations  and  sensitivity  analyses  may  need  to  be 
conducted  to  get  the  best  representative  model. 

1.6  Methodology 

A  Value  Focused  Thinking  (VFT)  model  will  be  used  in  this  research  to  evaluate 
alternatives  (SABER  contractor  proposals)  using  a  value  hierarchy  and  decision  makers’ 
weighted  values  to  aid  in  the  decision  analysis  of  a  source  selection.  A  VFT  approach 
was  used  because  the  values  inherent  in  the  decision  process  are  the  priority  of  the  entire 
methodology,  as  opposed  to  the  standard  decision  process  of  focusing  on  the  alternatives. 
By  switching  the  attention  to  values,  the  decision  process  now  becomes  proactive  rather 
than  reactive.  Keeney  (1996)  states,  “A  shift  to  this  way  of  thinking  about  decisions  can 
significantly  improve  decision  making  because  values  guide  not  only  the  creation  of 
better  alternatives  but  the  identification  of  better  decision  situations.” 

This  research  will  be  conducted  in  a  two  phase  process.  The  first  phase  will  consist 
of  an  initial  questionnaire  presented  to  a  base  civil  engineering  organization  that  acted  as 
members  of  a  SABER  source  selection  committee  and  have  a  depth  of  knowledge  on 
SABER  contracts.  The  questionnaire  will  attempt  to  gain  further  insight  into  the  current 
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source  selection  process.  The  members  of  the  civil  engineering  organization  will  also  be 
exposed  to  the  VFT  methodology  and  take  part  in  the  development  of  a  VFT  model  to 
select  a  SABER  contractor.  Between  the  two  phases  of  the  research,  the  model  will  be 
developed  through  several  iterations. 

The  second  phase  of  research  will  present  a  final  model  and  the  results  of  analysis 
gained  form  incorporating  empirical  data  gathered  on  seven  contractors.  The  civil 
engineering  organization  will  then  complete  a  final  questionnaire  to  gain  insight  into  the 
usefulness  and  practicality  of  the  VFT  model.  These  two  phases  of  research  will  help 
perform  a  comprehensive  analysis  on  the  current  source  selection  process,  the  VFT 
process,  the  model’s  performance,  and  validation  and  insight  into  the  usefulness  of  the 
model. 

1.7  Assumptions  and  Limitations  of  Research 

The  assumptions  involved  with  this  research  effort  are  that  the  VFT  model  will 
provide  valid  forms  of  objective  analysis  to  the  SABER  source  selection  process. 
Additionally,  it  is  assumed  that  the  model  will  be  able  to  show  the  importance  of 
evaluating  and  analyzing  all  competitive  alternatives  and  will  show  the  flaws  that  stem 
from  only  evaluating  lowest  priced  contractors  in  the  selection  process. 

This  research  will  utilize  CE  personnel  involved  with  a  SABER  contract  at  one 
particular  Air  Force  base;  therefore,  further  research  may  have  to  be  perfonned  to 
incorporate  this  type  of  model  for  other  locations  and  decision  makers.  The  overall 
model  will  attempt  to  capture  universal  factors  affecting  the  selection  of  a  SABER 
contractor;  however  the  weightings  applied  to  the  values  within  the  model  and  value 


10 


function  structures  will  be  dependent  upon  the  decision  makers  that  utilize  the  model. 
Since  empirical  coefficient  data  on  the  contractors  cannot  be  shared,  notional  data  sets 
may  be  generated  to  illustrate  the  capabilities  of  the  model.  This  is  the  first  attempt  to 
create  a  model  and  simulate  a  SABER  source  selection  that  evaluates  all  competing 
contractors;  therefore,  the  focus  was  aimed  towards  the  technical  decision  makers  (CE 
personnel).  Further  research  may  also  be  performed  to  incorporate  the  thoughts  and 
impressions  of  Contracting  personnel. 

1.8  Document  Structure 

There  are  four  remaining  chapters  in  this  thesis.  Chapter  2,  Literature  Review,  will 
provide  a  detailed  background  of  SABER  and  the  SABER  source  selection  process  as 
well  as  a  presentation  of  the  existing  knowledge  base  regarding  Value  Focused  Thinking. 
Chapter  3,  Methodology,  will  illustrate  the  development  of  the  VFT  model  used  to 
evaluate  the  research  objective  specified  in  Chapter  1  as  well  as  explain  the  validation 
process  inherent  to  VFT.  The  chapter  will  also  analyze  the  results  of  the  questionnaire 
conducted  in  phase  one  of  the  research.  Chapter  4,  Data  Analysis,  will  discuss  and 
review  the  results  of  the  analysis  conduced  on  the  empirical  data  sets  of  the  seven 
contractors.  The  impact  of  the  analysis  on  a  SABER  source  selection  will  also  be 
discussed,  as  will  the  analysis  of  the  results  of  the  questionnaire  conducted  in  phase  two 
of  the  research.  Chapter  5,  Conclusions  and  Recommendations,  will  present  an  overall 
summary  of  this  research  effort  and  present  concluding  thoughts.  Future  research 
avenues  and  recommendations  for  implementation  of  the  model  will  also  be  presented. 
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Chapter  II.  Literature  Review 


This  chapter  provides  a  synopsis  of  recent  literature  regarding  Simplified 
Acquisition  of  Base  Engineer  Requirements  (SABER)  contracts  and  the  current 
procurement  process,  as  well  as  a  detailed  discussion  of  the  existing  knowledge  base 
representing  Value  Focused  Thinking  (VFT).  The  first  part  of  the  chapter  focuses  on 
multiple  aspects  of  SABER  contracting  to  include  the  purpose,  benefits,  and  the  source 
selection  process.  The  chapter  further  discusses  problems  that  arise  within  the  source 
selection  process,  which  is  the  motivating  factor  for  this  thesis  research.  The  second  part 
of  the  chapter  focuses  on  VFT  and  explains  the  ten-step  process  as  proposed  by  Shoviak 
(2001). 

2.1  Air  Force  Construction  Contracts 

The  Air  Force  has  two  main  avenues  that  are  utilized  to  execute  a  majority  of 
construction  projects.  These  are  Design  Bid  Build  (DBB)  and  SABER  contracts  (Henry 
and  Brothers,  2001).  The  Air  Force  is  constantly  changing  and  updating  its  facilities  and 
base  layouts  to  keep  pace  with  the  operational  changes  and  technological  advances  of  our 
cutting  edge  military.  Such  projects  may  include  renovating  buildings  for  new  training 
personnel  or  equipment,  demolishing  obsolete  structures  or  equipment,  constructing  or 
installing  state  of  the  art  structures  or  equipment,  abating  hazardous  materials,  changing 
building  configurations  for  different  mission  capabilities,  and  so  on.  Often,  time  is  a 
crucial  factor  for  implementing  these  small  scale  construction  projects.  The  DBB  method 
is  a  very  detailed  and  lengthy  process  requiring  100%  design,  drawings,  and 
specifications  that  contain  enough  detail  to  describe  the  construction  process  without  any 
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additional  explanations  (Henry  and  Brothers  2001).  This  process  can  also  be  very  costly 
as  architecture-engineering  (A/E)  firms  are  required,  and  the  design  will  usually  be  about 
10%  of  the  total  construction  cost  for  a  project.  With  this  level  of  detail  for  each  project, 
DBB  contracts  can  require  3  to  9  months  to  complete  just  the  design  phase  (Henry  and 
Brothers  2001);  therefore,  DBB  is  not  a  viable  option  for  the  small  scale,  time-critical 
construction  conditions  that  the  Air  Force  often  faces.  These  are  the  conditions  where 
SABER  proves  to  be  a  valuable  tool. 

In  contrast  to  DBB  contracts,  the  design  requirements  for  SABER  are  a  35% 
complete  (not  100%)  design.  This  greatly  reduces  the  amount  of  time  and  effort  required 
by  base  level  engineering  personnel,  thereby  expediting  the  project  start  time.  Once  the 
contract  is  awarded,  task  orders  can  be  developed,  processed,  and  work  started  in  as 
quickly  as  30  days  (Furr,  1996).  Not  only  can  a  SABER  project  be  completed  in  an 
extremely  timely  manner,  bypassing  much  of  the  contracting  administrative  work  of  DBB 
projects,  but  the  SABER  contractor  can  conduct  multiple  projects  simultaneously. 

2.2  SABER  Contracts 

Unlike  DBB  contracts  where  a  separate  contract  package  has  to  be  awarded  for 
each  project,  SABER  is  awarded  as  one  contract  which  will  include  one  basic  year  with 
four  option  years,  making  it  a  potential  five-year  contract.  During  this  time,  each  project 
to  be  completed  by  SABER  is  issued  as  a  task  order  (TO).  Because  there  is  a  unit  pricing 
guide  (UPG)  and  specifications  that  are  all  encompassing  in  the  original  SABER 
contract,  the  TO  packages  do  not  require  individual  specifications  and  cost  is  simply 
based  off  a  unit  pricing  guide.  The  coefficient  that  the  contractor  has  submitted  is  then 
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utilized  to  determine  the  final  price.  Coefficients  are  factors  multiplied  against  the 
standard  unit  prices  in  the  UPG  to  calculate  TO  prices  (IG5336.9201,  2005). 

2.2.1  Purpose  of  SABER 

As  stated  in  IG5336.9201  (2005),  the  purpose  of  the  Air  Force  SABER  program  is 
to  expedite  contract  award  of  civil  engineer  requirements  by  reducing  civil  engineer 
design  work  and  acquisition  lead-time.  SABER  is  to  be  utilized  for  non-complex,  minor 
construction  projects,  repair  work,  and  other  projects  that  require  minimal  design. 
SABER  can  be  utilized  for  larger  projects  with  proper  authority  approval  so  long  as  it  is 
in  accordance  with  the  Federal  Acquisition  Regulation  (FAR). 

2.2.2  Benefits  of  SABER 

The  benefits  of  SABER  include  improved  customer  service  and  responsiveness  and 
incentives  for  highly  motivated  contractors  to  produce  high  quality  work  in  a  timely 
manner  (IG5336.9201,  2005).  Once  a  SABER  contract  is  awarded,  the  contractor  will  be 
designated  a  location  on  base.  This  ensures  that  they  will  be  readily  accessible  for  issues 
that  may  arise  on  project  jobsites.  A  SABER  contractor  will  usually  have  several  jobs 
occurring  at  the  same  time,  so  availability  is  essential.  Air  Force  bases  also  have  an 
immense  demand  for  completed  construction  TOs.  The  SABER  contractor  realizes  that 
the  base  can  provide  a  large  amount  of  business  and  that  acquiring  future  delivery  orders 
depends  on  the  quality  and  timeliness  of  the  projects  they  complete.  Therefore,  the 
contractors  are  highly  motivated  to  turn  out  quality  work  in  a  timely  fashion  to  ensure 
future  business. 
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2.2.3  Source  Selection 


In  order  to  award  a  SABER  contract,  the  Government  must  implement  a 
competitively  negotiated  source  selection  process.  This  allows  the  competing  contractors 
to  have  equal  opportunity  when  bidding  for  the  contract,  and  allows  the  Government  to 
evaluate  each  offeror’s  proposal  based  on  price  and  other  nonprice-related  factors. 
“Source  selection  procedures  are  designed  to  (1)  maximize  competition;  (2)  minimize  the 
complexity  of  the  solicitation,  evaluation,  and  selection  process;  (3)  ensure  the  impartial 
and  comprehensive  evaluation  of  proposals;  and  (4)  ensure  selection  of  the  source  whose 
proposal  is  most  advantageous  and  realistic  and  whose  performance  is  expected  to  best 
meet  stated  Government  requirements”  (Nash  et  ah,  1999). 

2.2.3. 1  Background 

The  process  of  detennining  the  SABER  contractor  is  conducted  by  a  source 
selection  committee  that  is  directed  by  a  Contracting  Officer  (CO);  the  committee  is 
comprised  of  decision  makers  from  the  civil  engineering  organization  (usually  consisting 
of  the  SABER  Chief,  Construction  Manager,  and  other  engineers  or  project  managers 
skilled  in  construction  practices).  A  SABER  Request  for  Proposal  (RFP)  states,  “Offeror 
shall  provide  infonnation  on  no  more  than  ten  (10)  of  the  most  recent  [in  the  past  three 
(3)  years]  contracts  (either  Federal,  State,  municipal,  or  commercial)  considered  most 
relevant  in  demonstrating  the  offerors  ability  to  perform  the  proposed  effort.”  (SABER 
RFP,  2004).  Furthermore,  it  also  states,  “Offerors  shall  provide  Attachment  #9,  Past  and 
Present  Perfonnance  Questionnaire  (Appendix  A),  to  those  agencies/firms  responsible  for 
the  solicitation  and  administration  of  those  identified  projects.”  These  performance 
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questionnaires  are  utilized  for  the  evaluation  process  of  the  offerors  as  described  in  the 
SABER  RFP  (2004): 

Using  questionnaires,  the  contracting  officer  will  seek  perfonnance 
information  based  on  (1)  the  references  provided  by  the  offeror  and  (2)  data 
independently  obtained  from  other  Government  and  commercial  sources. 

The  purpose  of  the  past  performance  evaluation  is  to  allow  the  Government 
to  assess  the  offeror’s  ability  to  perform  the  effort  described  in  this  RFP 
based  on  the  offeror’s  demonstrated  present  and  past  performance.  The 
contracting  officer  will  evaluate  the  past  perfonnance  of  the  seven  (7)  lowest, 
technically  acceptable  priced  offerors.  This  allows  for  efficiency  in  the 
source  selection  process. 

Once  all  proposals  have  been  received  by  Contracting  and  the  Contracting  Officer  has 
acquired  all  necessary  past  performance  and  background  information  on  the  offerors,  the 
decision  makers  will  begin  their  evaluation  process.  The  offerors  will  first  be  rated  on 
price  (i.e.,  the  coefficient  submitted)  in  rank  order  beginning  with  the  lowest-priced 
contractor. 

The  competing  contractors  submit  coefficient  information  which  will  be  multiplied 
by  each  future  TO  project  cost  based  on  the  UPG.  The  coefficients  “must  represent  all 
costs  associated  with  the  completion  of  the  requirements  of  the  contract  including,  but  not 
limited  to,  all  direct  costs,  overhead,  general  &  administrative,  bond  premiums,  profit, 
main  or  home  office  and  on  site  office  expenses”  (SABER  RFP,  2004).  Table  1  is  an 
example  from  the  SABER  RFP  demonstrating  how  offerors’  coefficients  can  be  weighted 
and  summed  to  detennine  an  overall  value  for  evaluation  purposes. 
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Table  1.  Coefficient  Ranking 
(SABER  RFP,  2004) 


CLIN* 

SAMPLE 

COEFFICIENT 

ACTUAL 
ASSESSED  % 

TOTAL 

0001 

1.17 

96% 

1.1232 

0002 

1.23 

3% 

0.0369 

0003 

1.21 

.5% 

0.0060 

0004 

1.27 

.5% 

0.0063 

Overall  Coefficient  Factor  for  Evaluation  Purposes 

1.1724 

*  CLIN  =  Contract  Line  Item 


After  rank  ordering  the  contractor  based  on  the  coefficient  infonnation  submitted, 
the  decision  makers  are  provided  an  evaluator  worksheet  (Appendix  B)  and  asked  to 
independently  rate  the  lowest  price  offeror.  The  decision  makers  then  collectively  rate 
the  offeror  with  another  evaluator  worksheet.  In  all  cases,  the  ratings  coincide  with  the 
definitions  as  described  in  Table  2.  If  the  lowest  price  offeror  receives  an  “Exceptional” 
rating,  that  offeror  will  be  awarded  the  contract.  If  the  lowest  price  offeror  receives  a 
“Very  Good”  or  below  rating,  the  selection  committee  moves  on  to  the  next  lowest  price 
offeror  and  repeats  the  process.  This  is  done  until  an  “Exceptional”  rated  offeror  is 
discovered.  If  the  “Exceptional”  rated  offeror  is  not  the  lowest  bidder,  an  analysis  will  be 
done  to  determine  the  best  value  contractor  for  the  Air  Force.  The  rational  for  this  type 
of  source  selection  is  based  on  the  guidelines  of  a  Lowest  Priced  Technically  Acceptable 
(LPTA)  source  selection  process. 
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Table  2.  Past  Performance  Ratings 
(Air  Force  Mandatory  Procedure  5315.3,  2005) 


Rating 

Definition 

Exceptional  /  High 
Confidence 

Based  on  the  offeror’s  performance  record, 
essentially  no  doubt  exists  that  the  successful  offeror 
will  successfully  perfonn  the  required  effort. 

Very  Good  / 

Significant  Confidence 

Based  on  the  offeror’s  perfonnance  record,  little 
doubt  exists  that  the  successful  offeror  will  perform 
the  required  effort. 

Satisfactory  / 

Confidence 

Based  on  the  offeror’s  performance  record,  some 
doubt  exists  that  the  successful  offeror  will  perform 
the  required  effort. 

None  /  Neutral 
Confidence 

No  performance  record  identifiable. 

Marginal  /  Little 
Confidence 

Based  on  the  offeror’s  performance  record, 
substantial  doubt  exists  that  the  successful  offeror 
will  perform  the  required  effort. 

Unsatisfactory  /  No 
Confidence 

Based  on  the  offeror’s  perfonnance  record,  extreme 
doubt  exists  that  the  successful  offeror  will  perform 
the  required  effort. 

2.2.3.2  Best  Value 

As  observed  from  the  selection  process  previously  described,  the  Government  may 
not  always  select  the  lowest  priced  bidder.  The  best  value  contractor  may  have  a  higher 
priced  bid,  but  it  is  determined  by  the  Government  that  the  difference  in  cost  is 
outweighed  by  the  quality  of  the  contractor  that  will  be  acquired.  “Best  value 
procurements  focus  on  selecting  the  contractor  with  the  offer  most  advantageous  to  the 
government,  price  and  other  factors  considered”  (Gransberg  and  Ellicott,  1996). 
Gransberg  and  Ellicott  provide  the  background  to  this  method  of  procurement  with  the 
following: 


18 


Beginning  in  1984  with  the  Competition  in  Contracting  Act  (CICA),  the  US 
congress  recognized  the  need  for  improved  procurement  procedures.  Federal 
acquisition  regulations  (FAR),  developed  to  implement  CICA,  include 
language  permitting  quality-  or  value-based  selections  (1996). 

The  need  for  greater  emphasis  on  the  overall  value  of  a  contractor  versus  just  focusing  on 

price  is  reiterated  by  Kashiwagi  and  Savichy  (2002): 

The  low-bid,  design-bid-build  construction  delivery  system  has  been  the 
standard  delivery  system  for  the  last  30  years.  In  the  last  ten  years, 
construction  nonperformance  problems  with  the  low-bid  system  have 
encouraged  owners  to  move  to  alternate  delivery  systems  such  as 
performance  contracting,  indefinite  delivery,  indefinite  quantity  (IDIQ), 
design-build  (DB),  and  construction  management  at  risk  (CMAR). 

2.2.3.3  Evaluation  Criteria 

The  evaluation  criteria,  to  include  factors  and  significant  subfactors,  utilized  in 
awarding  a  SABER  contract  must  be  described  within  the  solicitation  or  RFP  (FAR 
15.101-2,  2005).  Noted  earlier,  the  Government  will  evaluate  the  proposals  based  on 
price  and  nonprice-related  factors.  However,  nonprice-related  factors  are  encompassed 
into  a  past  performance  evaluation;  therefore,  the  factors  utilized  in  a  SABER  source 
selection  are  price  and  past  performance. 

The  1994  Federal  Acquisition  Streamlining  Act  (FASA),  signed  into  law  by 
Congress  on  October  13,  1994,  acknowledged  that  it  is  appropriate  and  relevant  for  the 
Government  to  consider  a  contractor’s  past  performance  in  evaluating  whether  that 
contractor  should  receive  future  work  (IG53 15.305,  2005).  Adversely,  FAR  15.304 
(2005)  notes:  “Past  perfonnance  need  not  be  evaluated  if  the  contracting  officer 
documents  the  reason  past  performance  is  not  an  appropriate  evaluation  factor  for  the 
acquisition.”  However,  there  is  currently  no  literature  that  can  produce  evidence  of  a 
source  selection  committee  not  utilizing  past  perfonnance  for  a  SABER  source  selection. 
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Price  is  a  straight  forward  factor  to  evaluate,  especially  for  SABER  since  the  only 
pricing  issue  is  the  coefficient  rate  (described  in  Section  2.3).  Therefore,  the  offerors  can 
be  rank  ordered  by  price  from  lowest  to  highest  immediately.  What  about  past 
performance?  There  has  been  a  trend  toward  using  past  performance  as  the  only  non-cost 
evaluation  factor  (Nash  et  ah,  1999).  Can  a  past  performance  factor  be  easily  assessed  as 
well?  According  to  Nash  et  al.  (1999),  Edwards  stated  that  past  performance  is  “used  to 
assess  an  offeror’s  capability,  comprising  of  three  elements:  (1)  observations  of  the 
historical  facts  of  a  company’s  work  experience  -  what  work  it  did,  when  and  where  it 
did  it,  whom  it  did  it  for,  and  what  methods  it  used;  (2)  qualitative  judgments  about  the 
breadth,  depth,  and  relevance  of  the  experience  base  on  those  observations;  and  (3) 
qualitative  judgments  about  how  well  the  company  perfonned,  also  based  on  the 
observations.”  For  one  evaluation  factor,  that  is  a  lot  of  information  to  interpret  and 
decipher  accurately. 

In  addition  to  the  Air  Force  guidance  governing  SABER  contracts,  other  criteria 
may  be  useful  to  explore.  In  fact,  numerous  researchers  have  identified  various 
categories  and  information  related  to  selection  criteria.  The  results  of  their  research  are 
summarized  in  the  following  paragraph. 

All  encompassing  criteria  were  noted  to  include  general,  technical,  managerial  and 
financial  criteria.  Further  broken  down,  the  stability  and  capability  aspects  were  explored 
concerning  the  financial  and  managerial  criteria,  and  an  in  depth  look  was  taken  at 
organizational  strength,  and  experience  of  comparable  construction.  Other  issues  that 
were  addressed  are  relevance  of  experience,  size  of  firm,  and  ones  safety  record.  All 
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these  criteria  have  been  researched  and  are  considered  significant  when  selecting  a 
construction  contractor  (Bransberg  and  Ellicott  1997). 

Gransberg  and  Ellicott  (1997)  also  explain  that  source  selection  panels  often 
develop  detailed  evaluation  criteria  capable  of  discriminating  between  various  proposals 
after  an  initial  review  of  all  proposals.  Criteria  can  include: 

•  Technical  excellence 

•  Management  capability 

•  Financial  capability 

•  Personnel  qualifications 

•  Prior  experience 

•  Past  performance 

•  Optional  features  offered 

•  Completion  date 

•  Risk  to  the  government 

This  represents  a  much  larger  list  than  the  Air  Force’s  practice  of  basing  decision  on  just 
two  criteria  (price  and  past  perfonnance).  Stated  previously  by  Nash  et  al.  (1999),  there 
are  several  subfactors  that  can  be  considered  with  past  performance  alone.  There  is  an 
even  greater  potential  for  many  more  underlying  subfactors  to  be  considered  in  the 
criteria  list  provided  by  Gransberg  and  Ellicott. 

2.2.4  SABER  Source  Selection  Problems 

With  a  large  list  of  potential  criteria  to  base  a  best  value  SABER  contractor  on,  the 
fact  that  the  Air  Force  is  only  utilizing  price  and  past  perfonnance  seems  to  negate  the 
intent  of  the  source  selection  process.  Additionally,  if  all  the  factors  and  subfactors  are 
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not  fully  explored  and  understood  by  the  decision  makers  and  the  bidding  contractors, 
how  effective  and  efficient  is  the  process  really  going  to  be?  The  decision  makers  are 
relying  totally  on  subjective  information  to  make  an  evaluation  and  cannot  objectively 
pinpoint  differences  that  may  be  evident  between  the  contractors.  Additionally,  the 
contractors  are  not  fully  aware  of  all  of  the  criteria  that  the  decision  makers  value  because 
they  are  not  presented  in  the  RFP.  “Prior  to  submitting  any  contract  proposal,  companies 
are  always  well  served  to  conduct  the  upfront  research  necessary  to  understand  a 
government  agency’s  most  important  requirements,  which  are  not  always  readily 
apparent  in  an  RFP”  (Kennedy  and  Cannon,  2004).  To  add  to  the  confusion,  this  whole 
process  needs  to  be  accurately  documented  and  justified.  “They  [agency  contracting 
officials]  argue  that  evaluations  tend  to  be  inflated,  with  past  performance  scores  offering 
little  real  discrimination  among  candidates”  (Burman,  1997).  This  trend  is  supported  by 
Kelman  (2005),  “At  a  conference  I  attended  recently,  a  participant  complained  that 
vendors’  past-performance  evaluations  look  alike,  making  it  hard  for  them  to  be  a 
differentiator  in  source  selection.” 

Offerors  that  lose  out  on  the  contract  have  the  right  to  see  how  they  rated  in  the 
process  and  can  also  challenge  the  final  decision.  “Thus,  the  fair  and  evenhanded 
evaluation  of  past  performance  has  become  a  critical  necessity,  and  there  have  been 
numerous  recent  protests  challenging  past  performance  assessments”  (Nash  et  ah,  1999). 
Thus,  the  issue  of  proper  documentation  becomes  critical.  The  Assistant  Secretary  of  the 
Air  Force  stated,  “While  the  Source  Selection  Authority  (SSA)  has  broad  discretion  in 
making  the  best  value  source  selection  decision,  the  decision  must  be  consistent  with  the 
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evaluation  factors  in  the  solicitation.  SSAs  must  fully  document  the  rationale  for  their 
decisions  in  Source  Selection  Decision  Documents  (SSDDs)”  (Sambur,  2003). 

FAR  15.305  (2005)  states  that  “evaluations  may  be  conducted  using  any  rating 
method  or  combination  of  methods,  including  color  or  adjectival  ratings,  numerical 
weights,  and  ordinal  rankings.”  Smith  (2004)  notes  that  “color  or  adjectival  ratings  fail  to 
provide  a  mechanism  for  evaluation  teams  to  weigh  each  criteria  with  relation  to  the  other 
factors  making  it  cumbersome  to  detennine  best  value.”  SABER  Contracts  are  awarded 
on  the  basis  of  only  two  criteria,  price  and  past  perfonnance,  with  very  little  guidance 
regarding  the  award  process  of  the  source  selection.  This  combination  is  a  double  threat 
in  that  the  tools  and  directions  for  a  SABER  source  selection  are  not  well  established  and 
defined. 

The  SABER  source  selection  process  should  be  a  non-complex,  impartial, 
objective,  and  easily  documented  process.  However,  without  a  Finn  understanding  of  the 
factors  that  are  utilized  in  the  selection  process,  the  process  will  become  frustrating, 
confusing,  and  misguided. 

The  source  selection  decision  is  by  nature  a  subjective  one,  even  when  the 
evaluation  standards  are  made  as  objective  as  possible.  The  evaluations 
demand  that  judgment  and  tradeoffs  must  be  made  (usually  between  cost, 
performance,  and  risk).  In  Air  Force  source  selections,  relative  weights  of 
the  criteria  are  usually  established  and  communicated  in  the  request  for 
proposal  -  but  these  are  usually  not  quantified,  either  in  the  proposal  or  in 
the  actual  source  selection.  Thus,  judgmental  factors  in  the  evaluation  of 
the  proposals,  the  comparison  of  proposals  one  against  another,  and  the 
decision  by  the  source  selection  authority  suggest  that  the  process  will  by 
nature  have  considerable  subjectivity,  especially  given  the  complex  nature 
of  the  acquisitions”  (Templin  and  Noffsinger,  1994) 
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2.3  Value  Focused  Thinking 

Selection  of  a  SABER  contractor  is  an  in-depth  process  of  gathering,  evaluating, 
and  interpreting  infonnation.  As  discussed  earlier,  the  SABER  source  selection  process 
is  ill-defined  and  there  is  currently  no  way  to  alleviate  the  subjectivity  of  the  process; 
therefore,  there  is  a  need  for  some  form  of  standardized  clarification  to  the  directional 
guides.  A  structured  decision  making  process  can  aid  in  fully  understanding  a  decision 
being  made  as  well  as  help  in  choosing  the  best  alternative  (Duncan,  2004).  With  today’s 
technological  capabilities,  computer  assistance  is  a  mere  matter  of  application.  “During 
the  past  decade,  millions  of  business  people  discovered  that  one  of  the  most  effective 
ways  to  analyze  and  evaluate  decision  alternatives  involves  using  electronic  spreadsheets 
to  build  computer  models  of  the  decision  problems  they  face”  (Ragsdale,  2004). 

Most  decision-making  processes  focus  on  alternatives  at  hand,  and  then  try  and 
choose  the  best  alternative  for  the  situation.  This  form  of  reasoning  is  restricted  because 
“it  is  reactive,  not  proactive”  (Keeney,  1996).  Referring  to  this  logic  as  alternative 
focused  thinking,  Keeney  (1996)  states  that  the  decision  maker  concentrates  first  on 
alternatives  and  only  afterwards  addresses  the  objectives  or  criteria  to  evaluate  the 
alternatives.  On  the  other  hand,  Value  Focused  Thinking  (VFT)  is  a  method  of  decision 
making  in  which  the  focus  is  on  values  (important  criteria,  preferences,  and  insightful 
information)  first  and  alternatives  only  after  the  values  or  criteria  for  the  decision  have 
been  fully  addressed.  VFT  “aids  in  articulating  and  using  your  fundamental  values  to 
guide  and  integrate  your  decisionmaking  activities”  (Keeney,  1996). 

VFT  has  been  utilized  in  several  corporate  applications  to  enhance  understanding 
and  provide  insight  into  decision  and  business  processes.  British  Columbia  Hydro; 
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Conflict  Management,  Inc.  (CMI);  Strategic  Decisions  Group  (SDG);  and  the 
Municipality  of  Metropolitan  Seattle  to  name  a  few  have  all  incorporated  aspects  of  the 
VFT  methodology  to  better  the  capabilities  of  their  organizations.  Shoviak  (2001) 
proposed  the  ten-step  VFT  model  which  is  detailed  in  the  following  sections  and  is  the 
foundation  for  the  research  model. 

2.3.1  Step  1  -  Problem  Identification 

Defining  the  problem  to  be  solved  is  a  seemingly  simplistic  step,  but  must  be 
addressed  carefully.  The  decision  makers  must  detennine  their  objectives  fully  and 
identify  all  desired  outcomes  completely.  Once  the  course  of  the  problem  is  wholly 
identified,  then  the  decision  makers  will  understand  the  intent  and  direction  of  the 
decision  making  process.  If  the  problem  is  misdirected  or  defined  improperly  from  the 
start,  the  analysis  process  will  likewise  be  misdirected  and  the  effort  will  be  futile.  A 
well  defined  objective  question  for  the  problem  will  harness  the  efforts  of  the  decision 
makers  in  the  proper  direction,  with  a  focused  and  meaningful  analysis  process. 

2.3.2  Step  2  -  Value  Hierarchy  Construction 

A  value  hierarchy  is  a  graphical  representation  of  the  values  that  the  decision 
makers  detennine  necessary  to  solve  the  decision  problem.  Value  hierarchies  allow  the 
decision  makers  a  visual  of  how  their  values  fall  within  the  decision  process  as  well  as 
how  the  values  may  influence  the  overall  decision.  Value  hierarchies  usually  present  a 
tree-like  structure  with  the  overall  decision  problem  (objective)  as  the  sole  top  element 
and  layers  or  tiers  of  values  branching  off  below.  These  values  in  turn  branch  down  to 
definitive  evaluation  measures  or  measuring  scales  for  the  degree  of  attainment  of  an 
objective  (Kirkwood,  1997). 
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The  overall  values  and  measures  are  determined  by  the  decision  makers;  however, 
there  are  several  techniques  to  help  guide  the  building  process  for  the  value  hierarchy. 

The  most  common  are  bottom-up  and  top-down  value  structuring  (Kirkwood,  1997).  The 
bottom-up  technique  is  when  values  and  measures  are  developed  from  existing 
alternatives.  The  measures  are  then  iteratively  grouped  to  form  tiers  within  the  hierarchy 
until  the  overall  encompassing  objective  is  defined.  The  top-down  approach  does  not 
have  clearly  defined  alternatives  up  front;  therefore,  it  defines  the  objective  first  and 
develops  subsequent  values  that  are  detennined  important  to  the  decision.  The  values  are 
then  iteratively  broken  down  to  clearly  defined  evaluation  measures.  VFT  models  are 
typically  developed  utilizing  the  top-down  technique  (Kirkwood,  1997). 

Another  method  is  the  “gold  standard”  and  involves  the  review  of  literature  that  is 
relevant  to  the  overall  objective  (Weir,  2005).  Relevant  literature  may  include 
regulations,  guides,  laws,  etc.  to  generate  valid  criteria  when  considering  the  objective. 
This  process  will  aid  in  utilizing  values  that  have  already  been  developed  by  the 
organization,  or  other  entity;  therefore,  it  will  be  easier  to  justify  and  defend. 

Generating  values  can  also  be  accomplish  by  using  the  “silver  standard,”  which 
involves  talking  to  relevant  personnel  who  hold  a  vested  interest  in  the  objective  and  is 
often  referred  to  as  casual  empiricism  (Weir,  2005;  Kirkwood,  1997).  This  method  is 
usually  accomplished  by  holding  large  group  sessions.  The  groups  will  consist  of 
decision  makers,  subject  matter  experts,  and  others  who  may  offer  valuable  insight  into 
the  value  creation  process.  A  typical  technique  is  to  list  ideas  individually  first,  and  then 
brain-storm  as  a  group  (Weir,  2005).  This  will  open  all  avenues  for  value  creation 
instead  of  focusing  on  one  thought  process.  Obtaining  values  from  the  stakeholders  not 
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only  provides  legitimate  inputs,  but  it  also  creates  “buy-in”  from  the  relevant 
stakeholders  since  they  know  their  values  are  being  considered  (Kirkwood,  1997). 

An  additional  value-generating  technique  is  the  “platinum  standard”  (Weir,  2005). 
This  method  involves  a  combination  of  looking  at  relevant  vision  statement,  objectives, 
and  doctrinal  goals  along  with  interviews  of  technical  experts  and  leadership  officials. 
“This  method  will  usually  produce  the  most  insightful,  simple,  and  logical  structure  for 
the  value  hierarchy”  (Duncan,  2004). 

To  aid  in  visualization  of  what  a  value  hierarchy  looks  like,  an  example  developed 
by  Duncan  (2004)  will  be  utilized  in  which  the  fundamental  objective  is  to  by  the  best 
truck  (Figure  1).  The  first  tier  of  this  example  represents  the  values  that  the  decision 
maker  considers  the  most  important  (cost,  perfonnance,  and  appearance).  Performance 
has  been  further  broken  down  to  a  second  tier  of  values  (power  and  off-road  capability). 
The  example  progression  of  this  model  developed  by  Duncan  will  be  utilized  to  aid  in  the 
description  of  the  remaining  VFT  steps  within  this  chapter. 


27 


Figure  1.  Example  Value  Hierarchy  (Duncan,  2004) 


The  desirable  properties  for  a  value  hierarchy  include:  completeness, 
nonredundancy,  independence,  operability,  and  small  size  (Kirkwood,  1997). 
Completeness  of  a  value  hierarchy  ensures  that  all  evaluation  concerns  necessary  to  the 
outcome  of  the  decision  are  included.  Completeness  is  also  referred  to  as  collectively 
exhaustive.  Furthermore,  the  evaluation  measures  must  adequately  measure  how  each 
alternative  achieves  the  objective  based  on  the  given  measure  for  the  value  hierarchy  to 
attain  true  completeness.  If  the  value  hierarchy  is  not  complete,  then  false  information 
can  be  derived  from  the  final  analyses  due  to  any  lacking  information. 

Nonredundancy  means  that  no  value  or  measure  within  the  hierarchy  is  covered 
more  than  once.  Nonredundancy  is  also  referred  to  as  mutually  exclusive.  The 
importance  of  nonredundancy  is  evident  “if  more  than  one  evaluation  measure  indicates 
the  degree  of  attainment  for  a  particular  objective  (that  is,  the  evaluation  measures  are 
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redundant),  then  that  objective  will  probably  receive  more  weight  than  was  intended 
when  the  weights  were  assigned  to  the  various  evaluation  measures”  (Kirkwood,  1997). 

Independence  simply  means  the  score  given  to  one  measure  within  the  hierarchy 
has  no  affect  on  the  score  of  the  other  measures.  Duncan  uses  the  example  of  having  the 
measure  for  power  in  the  truck  example  as  horsepower  and  the  measure  for  off-road 
capability  as  acceleration.  These  measures  are  not  independent  because  a  higher 
horsepower  score  will  always  result  in  a  higher  acceleration  score.  The  score  for 
acceleration  is  dependent  on  the  score  for  horsepower  (Duncan,  2004). 

Operability  is  the  ability  of  the  users  to  be  able  to  easily  understand  and  effective 
use  the  hierarchy.  Ensuring  that  the  value  hierarchy  falls  within  reasonable  operability 
standard  is  a  direct  result  of  how  the  users  decide  what  is  most  efficient.  “In  practice,  it 
may  be  necessary  to  compromise  with  respect  to  some  of  the  other  desirable 
characteristics  in  order  to  use  evaluation  measures  that  are  operable”  (Kirkwood,  1997). 

The  final  desirable  property  is  small  size.  A  smaller  value  hierarchy  is  easier  to 
communicate  to  the  end  users  and  among  the  decision  makers.  Ensuring  a  concise 
hierarchy  enables  more  efficient  evaluations  of  alternatives  as  time  and  effort  are  not 
consumed  by  an  overwhelming  amount  of  various  evaluation  measures.  The  key  to 
creating  a  complete  and  useful  value  hierarchy  “must  be  balanced  against  the  need  to 
finish  an  analysis  within  a  realistic  time  frame  and  budget”  (Kirkwood,  1997). 

2.3.3  Step  3  -  Evaluation  Measure  Development 

Evaluation  measures  must  be  developed  to  measure  the  degree  of  attainment  each 
value  will  contribute.  These  evaluation  measures  allow  for  an  unambiguous  rating  of 
how  well  the  alternatives  do  in  comparison  to  the  overall  objective  (Kirkwood,  1997). 
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Figure  2  illustrates  how  evaluation  measures  fit  into  the  value  hierarchy.  Values  can  be 
represented  by  a  single  measure  (color  for  appearance)  or  by  multiple  measures  (4WD, 
tire  size,  and  suspension  for  off  road  capability).  The  intent  is  to  develop  all  measures 
that  are  necessary  to  accurately  measure  how  the  alternatives  will  influence  the  values. 


Figure  2.  Example  Value  Hierarchy  with  Measures  (Duncan,  2004) 


Evaluation  measures  are  classified  into  different  scales  which  are  either  natural  or 
constructed,  and  either  direct  or  proxy  (Kirkwood,  1997).  A  natural  scale  is  one  that  is 
easily  understood  by  everyone  and  often  commonly  used.  Dollars  to  measure  a  price 
paid  is  an  example  of  a  natural  scale.  Constructed  scales  are  used  when  natural  scales  are 
not  suitable  or  available  to  measure  the  degree  of  attainment  for  a  particular  objective. 
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Constructed  scales  are  commonly  portrayed  categorically  to  encompass  a  finite  unit 
range.  In  reference  to  the  truck  example,  a  good  example  of  a  constructed  scale  is  the 
suspension  measure  under  the  off-road  capacity  value  where  the  categories  can  be 
defined  as  standard,  off-road,  and  towing  (Duncan,  2004).  The  differing  categories  are 
developed  to  meet  the  varying  decision  makers’  scores  for  each  alternative.  Not  only  do 
the  categories  need  to  be  developed,  but  they  also  need  to  be  clearly  defined  to  alleviate 
any  subjectivity  or  bias  from  the  decision  makers. 

Each  measure  scale  is  also  classified  as  direct  or  proxy.  “A  direct  scale  directly 
measures  the  degree  of  attainment  of  an  objective,  while  a  proxy  scale  reflects  the  degree 
of  attainment  of  its  associated  objective,  but  does  not  directly  measure  this”  (Kirkwood, 
1997).  Again,  dollars  to  measure  price  is  a  good  example  of  a  direct  scale.  Gross 
national  product  (GNP)  is  an  example  of  a  proxy  scale  (Kirkwood,  1997). 

Evaluation  measure  scales  can  be  presented  in  an  order  of  preference,  Figure  3, 
which  is  natural  direct,  constructed  direct,  natural  proxy,  and  constructed  proxy  (Duncan 
2004).  This  order  is  an  important  consideration  when  selecting  a  measure  scale  as  the 
more  highly  preferred  scales  will  provide  a  more  comprehensive  and  user-friendly 
decision  making  tool.  The  reason  for  this  is  because  if  a  decision  maker  is  presented  with 
a  scale  they  are  accustomed  to  using  such  as  dollars  (natural  direct),  they  will  have  an 
immediate  understanding.  Kirkwood  (1997)  explains  that  the  type  of  audience  must  be 
taken  into  consideration  when  developing  measure  and  scales.  A  less  technical  audience 
may  require  measures  that  are  not  as  technical  in  nature  but  rather  more  operable 
(Duncan,  2004).  Lastly,  to  ensure  that  the  scales  can  be  communicated  easily  and  are  not 
ambiguous,  the  scales  should  pass  the  clairvoyance  test  (Kirkwood,  1997).  As  described 
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by  Duncan  (2004),  Kirkwood  (1997)  asks,  “If  a  clairvoyant  were  available  who  could 
foresee  the  future  with  no  uncertainty,  would  this  clairvoyant  be  able  to  unambiguously 


assign  a  score  to  the  outcome  from  each  alternative?” 


Table  3.  Preferred  Order  of  Use  for  Evaluation  Measure  Scales 
(Duncan,  2004;  Weir,  2005) 


Natural 

Constructed 

Direct 

1 

2 

Proxy 

3 

4 

2.3.4  Step  4  -  Value  Function  Creation 

Single  Dimensional  Value  Function  (SDVFs)  must  be  defined  to  allow  the  scales 
defined  for  the  evaluation  measures  to  be  converted  to  a  common  scale  so  scores  can  be 
combined,  compared,  and  analyzed  (Duncan,  2004).  Since  there  are  multiple  evaluation 
measures  and  multiple  evaluation  units,  it  is  essential  to  have  to  have  a  common  scale  for 
consistency.  The  SDVFs  convert  the  units  of  measure  into  value  on  a  scale  from  zero 
(lowest  attainable  value)  to  one  (highest  attainable  value).  Therefore,  an  alternative  that 
has  the  least  preferred  score  for  a  given  measure  will  have  a  value  of  zero,  and  an 
alternative  that  has  the  most  preferred  score  for  a  given  measure  will  have  a  value  of  one 
(Kirkwood,  1997).  This  conversion  process  helps  turn  subjective  decisions  into 
quantifiable  scores  that  can  be  objectively  examined  by  decision  makers. 

“Each  individual  SDVF  is  defined  by  its  shape,  which  is  determined  by  soliciting 
input  from  the  decision  maker  of  subject  matter  experts  (SMEs)  to  detennine  how  the 
evaluation  measure  scores  should  be  converted  into  ‘value’  units”  (Duncan,  2004). 
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An  SDVF  must  be  monotonic,  meaning  that  no  two  scores  for  a  given  measure  can  be 
assessed  the  same  associated  value  amount.  Therefore,  SDVFs  are  either  positive 
(increasing)  or  negative  (decreasing)  across  the  range  of  the  function.  SDVFs  with  an 
increasing  function  will  always  prefer  higher  levels  over  lower  levels  for  a  measure. 
Inversely,  SDVFs  with  a  decreasing  function  will  always  prefer  lower  levels  over  higher 
levels  for  a  measure. 

The  shape  of  each  SDVF  is  determined  by  the  returns  to  scale  that  are  associated 
with  each  respective  measure  (Duncan,  2004).  The  returns  of  scale  are  split  into  four 
categories:  constant,  increasing,  decreasing,  or  any  combination  thereof.  There  are  two 
types  of  SDVF  shapes  that  represent  these  returns  of  scale:  piecewise  linear  and 
exponential  (Kirkwood,  1997).  Piecewise  linear  functions  are  used  when  there  is  a  finite 
number  of  scoring  levels  for  the  measure  being  considered.  Categorical  functions  are  a 
prime  example  of  a  piecewise  linear  function.  Exponential  functions  are  utilized  when 
there  are  infinite  or  continuous  scoring  levels  for  the  given  measure.  Examples  of  the 
SDVF  shapes  discussed  are  given  in  Table  4.  Specific  examples  in  relation  to  the  truck 
buying  example  are  presented  in  Figures  3  and  4,  where  the  color  measure  represents  a 
monotonically  increasing  piecewise  linear  function  and  the  cost  evaluation  measure 
represents  a  monotonically  decreasing  exponential  function. 


33 


Table  4.  Common  SDVF  Shapes  (Duncan,  2004) 
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Figure  3.  Monotonically  Increasing  Piecewise  Linear  SDVF  (Duncan,  2004) 
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Figure  4.  Monotonically  Decreasing  Exponential  SDVF  (Duncan,  2004) 
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2.3.5  Step  5  -  Value  Hierarchy  Weighting 

Once  the  values  and  measure  of  the  decision  maker  are  identified  and  clearly 
defined  and  the  evaluations  measures  created,  the  relative  importance  of  the  values  and 
measures  can  be  determined  through  the  process  of  weighting.  This  is  important  because 
the  values  that  the  decision  maker  deems  the  most  important  will  become  evident  as 
weighting  values  are  assigned.  Also,  weighting  the  value  hierarchy  will  ensure  that  the 
intended  weight  that  the  decision  maker  assigns  to  a  specific  value  or  measure  will  stay 
constant  throughout  the  decision  making  and  analysis  processes,  eliminating  any  bias  or 
other  discriminating  influences  during  the  course  of  a  decision.  Weights  are  assigned  and 
expressed  either  locally  or  globally. 

2.3. 5.1  Local  Weighting 

The  local  weighting  of  a  value  hierarchy  is  the  relative  importance  of  values  or 
measures  that  are  located  on  the  same  tier  or  level  and  same  branch  within  the  value 
hierarchy.  The  local  weights  of  all  the  values  or  measures  within  the  tier  of  any  given 
branch  of  the  hierarchy  will  sum  to  one.  Usually,  local  weighting  is  accomplished  with  a 
top-down  approach  through  the  hierarchy  using  either  the  direct  weighting  or  swing 
weighting  technique.  “The  direct  weighting  method  is  often  referred  to  as  the  100-marble 
weighting  system”  (Duncan,  2004).  The  decision  maker  is  asked  to  go  through  each 
branch  of  the  hierarchy,  and  at  each  tier  assume  that  they  possess  100  imaginary  marbles. 
The  decision  maker  will  then  divide  the  marbles  up  among  the  values  or  measures  to 
represent  the  amount  of  importance  that  the  decision  maker  feels  that  the  value  or 
measure  holds.  The  number  of  marbles  given  to  each  value  or  measure  is  then  divided  by 
100  to  give  the  proportion  of  value  the  decision  maker  has  for  the  values  and  measures  in 
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the  respective  branches  and  tiers.  Again,  the  local  weights  of  the  values  and  measures 
within  the  tier  of  any  given  branch  must  sum  to  one  (Figure  5).  The  100-marble 
technique  gives  the  decision  maker  a  clear  understanding  and  direct  approach  for 
weighting  values  and  measures.  They  can  instantly  visualize  and  adjust  the  weights  to 
reflect  their  preference. 


Figure  5.  Example  Hierarchy  with  Local  Weights  (Duncan,  2004) 


Swing  weighting  can  be  accomplished  using  two  variations  of  incorporating  the 
value  increments  of  the  evaluation  measures.  The  first  technique  addresses  the  value 
increment  of  the  evaluation  measures  from  the  least  preferred  end  to  the  most  preferred 
end,  then  scaling  each  of  these  value  increments  as  a  multiple  of  the  smallest  value 


37 


increment  (Kirkwood,  1997).  The  second  technique  is  similar  in  nature  but  has  the 
decision  maker  subjectively  determine  the  least  important  measure  within  that  tier  of  the 
branch  (Duncan,  2004).  The  decision  maker  will  then  determine  the  importance  of  the 
other  measures  in  multiples  of  the  previously  determined  least  important  measure.  The 
techniques  will  both  us  an  algebraic  summation  equation  which  will  sum  to  one  (the 
weights  within  each  tier  of  a  branch  must  sum  to  one)  to  determine  all  the  weighting 
values.  The  equation  will  be  in  terms  of  only  one  unknown  value,  the  least  important 
value  increment,  and  will  be  solved  for  that  unknown  value.  Once  the  unknown  factor  of 
the  equation  has  been  solved,  the  subsequent  weighting  values  within  the  equation  can  be 
determined. 

2.3.5.2  Global  Weighting 

Global  weighting  is  used  to  determine  the  overall  weight  of  each  value  or  measure 
in  relation  to  the  entire  value  hierarchy.  Global  weighting  enables  the  decision  maker  to 
see  how  each  value  and  measure  influence  the  overall  objective  to  ensure  that  the 
weighting  values  that  been  have  assigned  (utilizing  the  local  weighting  technique)  are 
valid.  Often,  the  global  weights  of  each  value  or  measure  are  determined  using  a  top- 
down  mathematical  approach  of  multiplying  the  local  weights  of  each  value  above.  An 
example  of  using  this  technique  for  the  truck  buying  example  is  shown  in  Figure  6.  To 
solve  for  the  global  weight  of  4WD,  its  local  weight  of  0.500  would  be  multiplied  by  the 
local  weight  of  the  value  of  off  road  capability  (0.600),  the  local  weigh  of  performance 
(0.300),  and  the  local  weight  of  the  overall  objective  (1.000).  The  resulting  global  weight 
of  4WD  is  therefore  0.090. 
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Figure  6.  Example  of  Hierarchy  with  Global  Weights 
in  Parentheses  (Duncan,  2004) 


2.3.6  Step  6  -  Alternative  Generation 

Once  the  valued  hierarchy  has  been  developed,  alternatives  can  be  generated.  By 
exploring  the  values  of  the  decision  makers  and  creating  a  value  hierarchy,  the  alternative 
generation  process  can  be  greatly  enhanced  by  providing  a  better  understanding,  more 
insight,  and  a  creative  open-minded  view  toward  the  types  of  alternatives  needed  to  solve 
the  decision  objective.  Many  times  the  decision  makers  will  limit  their  alternative 
selection  due  to  the  associative  process;  making  mental  associations  with  previous 
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situations  which  will  only  focus  their  minds  on  those  alternatives  which  reflect  similar 
qualities  (Kirkwood,  1997). 

One  technique  to  force  an  out-of-the-box  thinking  approach  to  alternative 
generation  is  to  use  a  strategy  table  (Kirkwood,  1997).  These  tables  identify  different 
essential  pieces  of  measures  or  alternatives  which  allows  the  user  to  develop  desirable 
alternatives.  Other  techniques  that  can  be  used  are  identifying  value  gaps  in  current 
alternatives  or  developing  a  best-case  scenario  alternative  by  aggregating  the  measures 
with  the  highest  scores  from  the  alternatives  (Weir,  2005).  A  value  gap  is  identified 
when  an  alternative  can  improve  its  rating  by  improving  a  certain  area.  By  improving 
these  areas,  new  alternatives  can  be  generated  to  exploit  these  value  gaps.  The  best-case 
scenario  develops  the  overall  best-case  alternative  by  using  the  best  values  of  all  the 
current  alternatives  collectively.  Screening  of  alternatives  may  be  utilized  if  too  many 
alternatives  have  been  identified  and  analysis  of  all  of  them  is  improbable  (Kirkwood, 
1997).  Also,  alternatives  may  already  be  presented  for  the  decision,  in  which  case  an 
alternative  generation  technique  may  not  be  required,  unless  deemed  necessary  for  further 
insight  into  the  decision  analysis  process. 

2.3.7  Step  7  -  Alternative  Scoring 

Once  the  alternatives  have  been  generated,  they  can  be  scored.  The  success  of  this 
process  is  greatly  dependent  upon  the  accurate  development  of  the  hierarchy  and  meeting 
the  criteria  discussed  in  previous  sections  of  this  chapter:  collectively  exhaustive, 
mutually  exclusive,  independence,  operability,  and  small  size.  If  the  hierarchy  has  been 
carefully  developed,  the  scoring  process  will  progress  with  relative  ease  and  accuracy. 

The  data  used  to  score  the  alternatives  can  be  a  very  labor  intensive  process  if  the  value 
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hierarchy  has  numerous  measures,  measures  with  difficult  to  obtain  data,  or  measures 
with  ambiguous  scoring  criteria  (Duncan,  2004).  To  ensure  proper  scoring  of  the 
alternatives,  there  are  three  rules  which  need  be  followed.  The  first  is  to  properly 
document  where  the  data  was  generated  for  each  alternative  so  that  the  scoring  can  be 
repeated  or  tested.  Second,  score  the  data  blindly  so  that  you  do  not  know  how  the  values 
will  influence  the  overall  outcome  of  the  decision.  And  lastly,  be  sure  to  score  one 
measure  at  a  time  across  all  alternatives  considered.  Following  these  steps  will  help 
ensure  a  consistent,  unbiased,  and  precise  scoring  process. 

2.3.8  Step  8  -  Deterministic  Analysis 

The  deterministic  analysis  is  developed  by  using  an  equation  combining  the  values 
of  each  alternative  obtained  by  the  scoring  data  translated  by  the  SDVFs,  with  the 
weights  detennined  by  the  decision  makers.  This  equation  is  called  the  additive  value 
function  and  forms  a  collective  overall  score  for  each  alternative  utilizing  the  cumulative 
paired  values  and  weights  of  the  measures.  These  overall  scores  can  then  be  utilized  to 
rack  and  stack  the  alternatives.  The  additive  value  function  requires  that  a  valid  SDVF  be 
associated  with  each  measure,  a  weight  be  assigned  to  each  measure,  and  the  summation 
of  the  weights  equals  one  (Duncan,  2004).  The  additive  value  function  can  be 
represented  as: 

n 

1=1 

where  v(x)  is  the  value  function  (the  overall  score  of  the  alternative),  v/yj,  is  the 
individual  measure  value  obtained  from  the  score  incorporated  with  the  corresponding 
SDVF,  and  2,  is  the  global  weight  of  the  measure  (Kirkwood,  1997). 
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2.3.9  Step  9  -  Conduct  Sensitivity  Analysis 

The  sensitivity  analysis  is  utilized  to  provide  the  decision  maker  further  insight  into 
the  analysis  of  the  decision  and  is  a  unique  function  of  this  model.  A  sensitivity  analysis 
allows  the  decision  maker  to  detennine  the  impact  on  the  rankings  of  alternatives  by 
making  changes  in  various  model  assumptions  (Kirkwood,  1997).  The  most  common 
form  of  sensitivity  analysis  is  to  analyze  how  different  weighting  values  would  influence 
the  results  of  the  deterministic  analysis.  “These  weights  represent  the  relative  importance 
that  is  attached  to  changes  in  the  different  evaluation  measures,  and  this  is  sometimes  a 
matter  of  disagreement  among  the  various  stakeholders  for  a  particular  decision” 
(Kirkwood,  1997).  One  way  to  explore  the  varying  weighting  preferences  of  decision 
makers  is  to  change  the  weight  of  one  value  while  holding  the  weights  of  all  the  other 
values  proportionally  constant  and  ensuring  that  all  value  weights  sum  to  one  (Kirkwood, 
1997).  Doing  so  enables  the  decision  makers  to  see  how  changes  in  weightings  can  affect 
the  rankings  of  alternatives.  By  incorporating  this  type  of  analysis,  an  objective  and 
systematic  approach,  accompanied  by  visual  graphs,  can  be  utilized  to  help  explain 
subjective  decision  processes. 

2.3.10  Step  10  -  Presentation  of  Results 

The  results  gained  from  the  model  will  not  only  be  rank  ordered,  but  will  be  easy  to 
present  and  discuss.  The  value  hierarchy  clearly  defines  and  illustrates  the  values  and 
measures  that  are  important  and  the  level  of  their  importance  through  the  globally 
weighted  values.  The  SDVFs  identify  the  manner  in  which  the  alternatives  were  scored, 
and  the  deterministic  analysis  again  clearly  illustrates  how  the  alternatives  rack  and  stack 
against  each  other.  The  sensitivity  analysis  adds  further  insight  as  to  how  the  alternatives 
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clearly  compare  with  one  another  and  what  specific  attributes  alternative  are  weak  or 
strong  on.  Also,  the  sensitivity  analysis  can  provide  insight  into  the  different  views  of 
decision  makers  and  their  corresponding  weighting  value  differences.  The  results  will  be 
developed  and  presented  to  answer  the  overall  objective  that  the  model  intended  to  help 
solve.  The  model  and  the  results  are  not  intended  to  make  a  final  decision;  that  is 
ultimately  up  to  the  decision  maker.  The  model  is  a  tool  to  be  utilized  to  gain  greater 
insight  into  the  decision  making  process  and  aid  the  decision  maker  in  a  thorough 
analysis  process.  Keeney  emphasizes  three  fundamental  ways  VFT  will  aid  in  a  decision 
process:  1)  to  recognize  and  identify  decision  opportunities,  2)  to  create  better 
alternatives  for  your  decision  problems,  and  3)  to  develop  an  enduring  set  of  guiding 
principles  for  your  organization  (Keeney,  1996). 
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Chapter  III.  Methodology 


Value  focused  thinking  (VFT)  can  be  utilized  to  determine  the  effectiveness  of  the 
Simplified  Acquisition  of  Base  Engineer  Requirements  (SABER)  source  selection 
process  by  exploring  all  the  values  that  the  Air  Force  deems  important  in  a  SABER 
contractor  and  converting  that  qualitative  data  to  firm  measurable  quantitative  data. 
Exploring  a  model  of  this  magnitude  not  only  will  enable  the  development  of  a 
hypothetical  ideal  SABER  contractor  to  perfectly  meet  the  needs  of  the  Air  Force,  but  it 
will  also  facilitate  comparative  analysis  of  real  world  alternatives.  Approaching  a 
decision  in  this  manner  immediately  eliminates  initial  biases  as  the  focus  and  efforts  of 
the  evaluation  team  have  been  put  into  the  needs  of  the  Air  Force  and  not  what  is  simply 
available  to  the  Air  Force. 

The  current  source  selection  process  of  Lowest  Price  Technically  Acceptable 
(LPTA)  explained  in  Chapter  1  may  have  been  sufficient  when  SABER  was  in  its  infancy 
and  considered  a  relatively  small  contract.  However,  SABER  now  represents  a 
substantial  effort  on  most  bases  with  numerous  task  orders  every  year.  The  process  of 
systematically  evaluating  all  qualified  contractors  and  selecting  the  one  that  represents 
the  best  value  to  the  government  is  a  complicated  one  in  which  many  competing 
objectives  must  be  considered.  As  explained  in  Chapter  2,  a  VFT  approach  was  used  in 
this  research  to  provide  a  decision  tool  for  the  process.  Since  it  would  have  been 
impractical  to  apply  this  approach  to  the  entire  civil  engineering  career  field,  one 
organization  was  selected  and  used  throughout  this  research  and  is  referred  to  as  the 
subject  organization.  This  organization  was  selected  because  of  their  recent  experience 
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in  selecting  a  SABER  contractor;  therefore,  the  members  of  the  organization  have  recent 
exposure  to  the  evaluation  process  and  evaluation  data  is  readily  available. 

The  incorporation  of  a  VFT  methodology  allows  the  values  and  measures  of  an  Air 
Force  SABER  contractor  to  be  identified,  weighted,  and  analyzed.  More  importantly,  the 
methodology  ensures  that  all  intended  weighting  values  remain  constant  throughout  the 
evaluation  process  (unless  intentionally  changed)  and  provides  a  method  to  compare  and 
delineate  between  all  competing  contractors.  The  VFT  methodology  provides  an 
objective,  accurate,  adaptable,  defendable,  and  quantifiable  analysis  to  aid  the  decision 
makers’  final  selection.  This  chapter  explains  how  Steps  1  through  7  of  the  VFT  process 
were  applied  to  develop  a  SABER  source  selection  model  to  provide  an  objective  and 
quantifiable  analysis  tool. 

3,1  Step  1:  Problem  Identification 

The  first  step  of  the  VFT  10-Step  methodology  is  to  identify  the  problem  and  then 
state  it  in  the  form  of  an  objective  question.  The  problem  identification  for  a  SABER 
source  selection  is  inherently  present  in  the  RPF:  select  the  contractor  that  represents  the 
best  value  to  the  government.  More  specifically,  the  contractor  selection  is  “based  on  the 
contractor  that  has  the  best  capability,  capacity,  and  coefficient  mix,  as  determined  by  the 
contractor  selection  criteria”  (Henry  and  Brothers,  2001). 

Initial  investigative  questions  were  developed  and  presented  to  the  members  of  the 
subject  organization  who  were  directly  involved  with  SABER  contracts  (i.e.,  through 
source  selections,  negotiations,  design,  implementation,  and  construction  management). 
These  questionnaires  were  administered  prior  to  any  exposure  to  the  VFT  methodology  to 
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prevent  any  possible  biases.  The  results  collectively  show  that  the  following  areas  are  in 
need  of  improvement  with  the  current  source  selection  process. 

•  A  need  for  the  technical  evaluators  to  review  cost 

•  A  clearer  understanding  of  evaluation  criteria  and  an  objective  way  to 
evaluate  -  controlling  personal  influences  and  partial  bias 

•  A  method  to  ensure  data  is  reliable  -  or  be  able  to  detect  unreliable  data 

•  A  method  to  ensure  evaluation  process  is  only  conducted  once,  and  properly 
done  in  that  one  time  -  easily  executed  and  documented 

•  A  method  to  ensure  intended  weightings  are  applied  consistently 

•  A  method  that  will  allow  all  competing  contractors  to  be  evaluated 

•  A  method  to  easily  differentiate  between  contractors  with  similar 
capabilities 

The  investigative  questions  and  corresponding  answers  are  presented  in  Appendix  C. 
Answers  are  indicated  as  Subject  A,  Subject  B,  and  so  on  for  purposes  of  anonymity. 

Throughout  the  research  effort  and  in  discussions  with  members  of  the  subject 
organization,  it  was  detennined  that  the  source  selection  process  is  flawed  in  several 
areas  -  with  emphasis  on  a  need  to  evaluate  all  competing  contractors,  a  need  for  the 
entire  selection  committee  to  evaluate  price,  and  a  need  to  ensure  intended  weighting 
values  hold  constant  throughout  the  selection  process.  Therefore,  the  VFT  methodology 
was  presented  to  members  of  the  subject  organization  and  was  determined  appropriate  as 
a  model  to  provide  insight  into  an  alternative  source  selection  method.  The  overall 
objective  question  became:  “Which  SABER  Contractor  will  provide  the  best  value  to 
Columbus  Air  Force  Base:  Contractor  A,  B,  C,  D,  E,  F,  or  G?” 
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3.2  Step  2:  Create  the  Value  Hierarchy 

Once  the  problem  has  been  clearly  identified,  the  next  step  is  to  generate  the  values 
relevant  to  the  overall  objective  and  then  organize  them  into  a  value  hierarchy.  The 
initial  method  used  to  identify  the  values  was  the  “silver  standard”  in  which  values  were 
obtained  from  subject  matter  experts  and  decision  makers.  The  VFT  methodology  and 
purpose  of  values  was  explained  to  the  members  of  the  subject  organization.  Once  they 
had  a  clear  understanding,  they  brain-stonned  to  formulate  the  list  shown  in  Table  5. 

From  that  list,  the  values  of  price,  past  performance,  and  technical  capabilities  were 
determined  to  be  the  most  important  collectively  by  the  group.  The  initial  layout  of  the 
first  tier  of  the  hierarchy  was  developed  with  the  use  of  the  silver  standard  and  submitted 
to  the  decision  makers  within  the  subject  organization  for  their  approval  (Figure  7). 

To  provide  further  insight  into  the  values  necessary  for  a  SABER  contractor,  the 
“gold  standard”  method  in  which  pertinent  documents  are  referenced  was  implemented. 
The  RFP,  past  perfonnance  questionnaires,  SABER  Guidance  (Air  Force  Contracting 
Construction  Guide  IG5336.9201,  2005),  and  relevant  literature  pertaining  to 
construction  contractor  criteria  were  all  referenced.  Further  value  creation  was 
accomplished  with  implementation  of  the  “platinum  standard”  in  which  the  knowledge 
bases  gained  from  literature  and  personnel  are  combined.  There  was  close  interaction 
with  the  subject  organization  for  several  months  to  further  review  the  RFP,  and  the  values 
generated  through  the  previously  discussed  platinum  standard  were  used  to  develop  an 
initial  hierarchy.  The  initial  hierarchy  underwent  several  iterations  wherein  values  were 
either  moved  or  deleted  from  the  hierarchy.  The  final  value  hierarchy  consisting  of  12 
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values  is  shown  as  Figure  8  and  was  determined  to  capture  the  intent  of  selecting  a  best 
value  SABER  contractor. 


Table  5.  Potential  Values  Obtained  Collectively 


•  Technical  Capabilities 

•  Relevancy 

•  Time  /Recency 

•  Business  Plan 

•  Past  Performance 

•  Quality-Control 

•  Similar  Scope  Work 

•  Price 

•  Manager  /  Team  Capabilities 

•  On-site  Capabilities  /  Capacities 

•  Dollar  Value  of  Work 

•  Project  Time  Overruns 

•  Poor  Performance  Claims 

•  Corrective  Action  Taken  for  Time  /  Performance  Issues 

•  Compliance  with  Contract  Terms  and  Conditions 

•  Punch-list  Items  After  Final  Inspection 

•  Contractor  Performs  as  Prime  or  Subcontractor 
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Figure  7.  Initial  Tier  of  Value  Hierarchy 
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Figure  8.  Final  Value  Hierarchy 


As  stated  earlier,  the  first  tier  values  of  price,  past  performance,  and  technical 
capabilities  were  determined  to  be  the  most  important  values  for  the  Air  Force  in 
selecting  a  SABER  contractor.  This  configuration  closely  mirrors  the  values  of  the 
current  source  selection  process;  however,  the  current  process  includes  technical 
capabilities  under  past  performance,  with  price  and  past  performance  evaluated  by 
separate  entities.  The  VFT  hierarchy  clearly  separates  the  three  values  and  includes  each 
one  within  the  model  so  they  can  be  individually  recognized  and  evaluated. 

Price  is  an  important  value  for  this  model  because  it  is  not  normally  viewed  by  the 
members  of  the  subject  organization  comprising  the  source  selection  committee.  The 
price  is  determined  by  the  coefficient  value,  and  the  government  does  not  want  a 
contractor  with  a  coefficient  that  is  too  high.  However,  the  government  does  not  want  a 
contractor  with  a  coefficient  that  is  too  low  either.  The  coefficient  is  how  the  contractor 
recoups  their  overhead  costs  and  makes  their  profit.  If  the  contractor  submits  an 
extremely  low  coefficient  to  win  a  contract,  they  may  lose  money  in  the  long  run  and  not 
be  successful.  The  government  not  only  wants  a  best  value  contractor,  but  a  successful 
SABER  program. 

Past  performance  is  necessary  for  the  government  to  evaluate  and  accurately 
gauge  how  well  the  contractor  will  conduct  their  overall  business  activities.  The  amount 
of  confidence  instilled  in  the  contractor  is  determined  through  this  value.  Additionally, 
technical  capabilities  will  help  the  Government  detennine  applicable  management  and 
quality  control  characteristics.  By  considering  past  performance  and  technical 
capabilities  as  two  separate  values,  the  government  can  evaluate  contractors  in  a  more 
concise  manner.  Each  value  in  turn  has  lower  tier  values  to  specify  further  what  the  Air 
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Force  determines  important  in  evaluating  the  contractors,  leading  into  the  mutually 
exclusive,  collectively  exhaustive  measures.  A  definition  of  each  value  is  presented  in 
Appendix  D. 

3.3  Step  3:  Develop  Evaluation  Measures 

After  finalizing  the  value  hierarchy,  the  next  step  was  to  develop  the  evaluation 
measures.  The  measures  provide  the  VFT  model  the  capability  to  evaluate  an  alternative 
in  a  quantitatively  objective  manner.  Therefore,  the  measure  value  functions  must  be 
clearly  defined  to  eliminate  subjectivity  as  much  as  possible  throughout  the  evaluation 
process.  Again,  the  members  of  the  subject  organization  were  utilized  in  the 
development  of  the  measures.  These  measures  were  determined  to  best  represent  how  to 
achieve  the  values  in  the  hierarchy  while  maintaining  the  integrity  of  independence  and 
nonredundancy.  Measures  that  were  determined  important  but  not  incorporated  into  the 
hierarchy  were:  proximity,  8(a)  (small  business),  work  done  within  past  three  years,  and 
projects  worked  concurrently.  These  measures  were  left  out  of  the  model  because  they 
are  mandatory  considerations  required  of  all  contractors  submitting  a  proposal.  The 
complete  value  hierarchy  with  measures  is  shown  in  Figure  9. 
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Figure  9.  Final  Value  Hierarchy  with  Measures  (shaded) 


The  intent  in  creating  the  measures  was  to  achieve  as  many  natural,  direct  scales  as 
possible  and  as  few  constructed,  proxy  scales  as  possible  for  ease  of  use  and 
understanding.  However,  the  natural,  direct  characteristics  could  not  be  met  due  to  the 
SABER-specific  nature  of  the  measures.  After  careful  consideration,  all  measures  were 
determined  to  be  categorical  in  nature.  Therefore,  6  of  the  measures  were  constructed, 
direct  and  18  were  constructed,  proxy.  All  measures  are  summarized  in  Table  6.  The 
decision  maker  detennined  the  types  of  scales  to  be  used  (categorical)  along  with  upper 
and  lower  bounds.  Examples  of  categories  and  category  definitions  for  the  “In-house  / 
Subcontract-out”  measure  and  “Cooperative”  measure  are  presented  in  Tables  7  and  8. 
The  definitions  for  each  of  the  measures  are  explained  in  Appendix  E. 
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Table  6.  Evaluation  Measures  Summary 


First  Tier 
Value 

Second  Tier 
Value 

Third  Tier 
Value 

Measure 

Lower  Bound 

Upper  Bound 

Price 

Coefficient 

1.15 

Past 

Performance 

Competency 

On-Site 

Capabilities 

In-house  / 
Subcontract-out 

Both 

Customer 

Satisfaction 

Rehire 

No 

Yes 

Cooperative 

Exceptional 

Contracting  Actions 
Against 

More  than  6 

0 

Relevancy 

<3 

5  < 

Size  $  Amount 

500  K< 

Military  Installation 
Experience 

No 

Yes 

Timely 

Performance 

Met  Performance 

Standards 

Exceptional 

14  Days  Late  < 

On  Time 

Resolve  Delays 

None 

Exceptional 

Completion  of 
Punch-list  Items 

Exceptional 

Warranty  in  Timely 
Manner 

14  Days  Late  < 

On  Time 

Technical 

Capabilities 

Labor 

Standards 

Compliance  w/  Laws 
and  Regs 

Exceptional 

Safety 

Standards 

Safety  Plan 

Exceptional 

Management 

Effectiveness 

Contract 

Requirements 

Exceptional 

Ability  to  Reduce 
Problems 

Exceptional 

On-site  Presence 

Exceptional 

Quality 

Control 

Quality  Control  Plan 

None 

Exceptional 

Quality 

Workmanship 

Exceptional 

Materials  Used  / 

Specified 

Exceptional 

Adequate  Submittals 

None 

Exceptional 

Compliance  w/  Regs 
and  Code 

Exceptional 
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Table  7.  Category  Definitions  for  In-house  /  Subcontract-out  Measure 


Category 

Category  Definition 

Neither 

Proposal  does  not  demonstrate  the  capacity  of  contractor  to 
effectively  conduct  construction  work  within  their  corporation 
nor  the  ability  to  effectively  acquire  and  manage 
subcontractor(s)  to  perform  construction  work. 

Subout  only 

Proposal  only  demonstrates  capacity  of  contractor  to 
effectively  acquire  and  manage  subcontractor(s)  to  perform 
construction  work. 

In-house  only 

Proposal  only  demonstrates  capacity  of  contractor  to 
effectively  conduct  construction  work  within  the  means  of 
their  corporation. 

Both 

Proposal  demonstrates  the  capacity  of  contractor  to  effectively 
perform  construction  work  either  by  the  means  of  their  own 
corporation  or  through  the  use  of  subcontractor(s). 

Table  8.  Category  Definitions  for  Cooperative  Measure 


Category 

Category  Definition 

Not 

Cooperative 

Proposal  demonstrates  negative  business  conduct  or  does  not 
address  cooperative  capabilities  at  all. 

Satisfactory 

Proposal  addresses  cooperative  capabilities  but  demonstrates 
inconsistencies  of  both  negative  and  positive  conduct. 

Good 

Proposal  addresses  cooperative  capabilities  in  strictly  positive 
manner. 

Exceptional 

Proposal  addresses  outstanding  positive  cooperative 
capabilities  to  include  incentives  exercised  to  maintain  or 
improve  capabilities. 
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3.4  Step  4:  Create  Value  Functions 

The  next  step,  and  the  most  difficult  in  this  research  effort,  was  to  create  the  value 
functions.  The  purpose  of  creating  the  value  functions  is  to  be  able  to  convert  the  actual 
scores  assigned  to  the  measures  to  a  corresponding  value  which  represents  the  preference 
of  the  decision  maker.  This  is  determined  by  developing  single  dimensional  value 
functions  (SDVFs)  for  each  measure.  The  x-axis  of  the  SDVFs  were  determined  in  the 
previous  step;  this  step  will  determine  the  corresponding  v-axis  values  for  each  category 
element  within  their  respective  measure.  The  y-axis  will  always  range  from  a  value  of  0 
(least  preferred)  to  a  value  of  1  (most  preferred)  to  represent  the  full  range  of  the  decision 
maker’s  value  spectrum.  The  upper  and  lower  bounds  of  the  measures  shown  in  Table  6 
are  the  equivalent  zero  (lower)  and  one  (upper)  values  on  the  SDVFs.  Completing  this 
step  is  necessary  to  convert  the  subjective  nature  of  the  evaluation  process  into  qualitative 
data.  This  allows  the  objective  analyses  to  be  conducted  later  on  in  the  modeling  process. 

Since  all  of  the  measures  for  this  model  are  categorical  in  nature,  the  decision 
maker  directly  assigned  values  to  the  discrete  functions.  This  process  was  difficult  and 
took  several  iterations  due  to  its  subjective  nature.  The  decision  maker  tended  to  rate 
different  categories  within  the  same  measure  as  the  same  value.  After  an  explanation  that 
an  exact  value  rating  for  two  different  categories  within  the  same  measure  is  essentially 
equating  those  categories,  the  decision  maker  quickly  changed  the  preference  values  to 
ensure  a  differentiation  between  the  categories.  Figure  10  shows  the  discrete, 
monotonically  increasing  SDVF  for  the  “Cooperative”  measure  as  an  example.  The 
SDVFs  for  the  remaining  measures  are  included  as  Appendix  E. 
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Figure  10.  SDVF  for  Cooperative  Measure 
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3.5  Step  5:  Weight  the  Value  Hierarchy 

Once  the  SDVFs  have  been  created,  the  next  step  is  to  weight  the  values  and 
measures  within  the  hierarchy.  Global  weighting  ultimately  determines  the  level  of 
importance  assigned  to  each  value  and  measure  in  relation  to  the  entire  hierarchy.  For  the 
purpose  of  this  research,  the  SABER  Chief  and  one  other  CE  representative  (who  also 
was  part  of  a  recent  SABER  source  selection  committee)  determined  the  weights  of  the 
values  and  measures  using  a  top-down  approach  with  the  100-marble  method.  Once  they 
agreed  upon  the  weight  values,  the  local  weights  were  used  to  calculate  the  global 
weights.  Figure  1 1  shows  the  local  and  global  weights  applied  to  the  value  hierarchy  in 
its  entirety. 

Table  9  provides  the  local  and  global  weights  for  each  measure  (sorted  in 
descending  order)  along  with  the  cumulative  global  weights.  The  measure  “Coefficient” 
holds  a  significant  amount  of  weight  (0.330).  This  is  to  be  expected  since  price  is  always 
going  to  be  a  significant  factor  in  selecting  a  contractor.  This  also  meets  the  standards  set 
forth  in  the  RFP  which  ensures  that  non-pricing  factors  will  significantly  outweigh  the 
pricing  factor.  “Past  Performance”  and  “Technical  Capabilities”  cumulatively  hold  a 
weight  of  (0.670)  which  significantly  outweighs  “Price”  at  a  weight  of  (0.330). 

The  next  set  of  fairly  significant  measures  are  “Compliance  w/  Laws  &  Regs,” 
“Compliance  w/  Regs  and  Code,”  “Safety  Plan,”  and  “Contract  Requirement.”  These 
measures,  when  combined  with  the  “Coefficient”  measure,  make  up  54.9%  of  the  total 
value  of  the  model.  Therefore,  these  are  the  measures  that  the  decision  maker  considers 
the  most  important  in  the  selection  of  a  SABER  contractor.  Furthermore  the  top  13 
measures  represent  approximate  80%  of  the  model’s  total  value.  Therefore,  in 
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accordance  with  common  VFT  practices,  these  13  measures  can  be  considered  the 
significant  attributes.  Every  competing  contractor  is  expected  to  have  high  scores  within 
these  measures  as  they  represent  a  large  portion  of  what  the  decision  maker  values.  If  the 
competition  is  strong,  it  would  not  be  unusual  for  several  alternatives  to  be  rated  evenly 
based  on  these  measures.  The  bottom  1 1  measures  do  not  hold  significantly  high  global 
weights;  however,  their  importance  is  still  significant.  These  measures  are  still 
considered  necessary  by  the  decision  maker;  they  just  happen  to  not  be  rated  as  important 
as  the  top  measures.  What  makes  the  bottom  measures  significant  is  that  they  can  help  to 
differentiate  between  highly  competitive  alternatives.  If  two  or  three  alternatives  score 
evenly  throughout  the  top  13  measures,  the  bottom  1 1  measures  will  be  able  to  weed  out 
the  top  alternative. 
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Figure  11.  Value  Hierarchy  with  Local  and  Global  Weights  (Global  in  Parentheses) 


Table  9.  Cumulative  Chart  of  Measure  Global  Weights 


Measure 

Local 

Weights 

Global 

Weights 

Cumulative 

Global 

Weight 

Coefficient 

1.000 

0.330 

0.330 

Compliance  w/  Laws  &  Regs 

1.000 

0.066 

0.396 

Compliance  w/  Regs  &  Code 

0.040 

0.053 

0.449 

Safety  Plan 

1.000 

0.050 

0.499 

Contract  Requirements 

0.600 

0.050 

0.549 

Scope  Multidiscipline 

0.400 

0.041 

0.590 

Size  $  Amount 

0.400 

0.041 

0.631 

Cooperative 

0.500 

0.036 

0.667 

Met  Perfonnance  Standards 

0.250 

0.034 

0.701 

Resolve  Delays 

0.250 

0.034 

0.735 

Completion  of  Punch-list  Items 

0.200 

0.027 

0.762 

On-site  Presence 

0.250 

0.021 

0.783 

Contracting  Actions  Against 

0.300 

0.021 

0.804 

Quality  Control  Plan 

0.150 

0.020 

0.824 

Submittals  in  Timely  Manner 

0.150 

0.020 

0.844 

Quality  Workmanship 

0.150 

0.020 

0.864 

Military  Installation  Experience 

0.200 

0.020 

0.884 

Materials  Used  /  Specified 

0.150 

0.020 

0.904 

Adequate  Submittals 

0.150 

0.020 

0.924 

Warranty  in  Timely  Manner 

0.150 

0.020 

0.944 

Primary  /  Subcontractor 

0.500 

0.015 

0.959 

In-house  /  Subcontract  out 

0.500 

0.015 

0.974 

Rehire 

0.200 

0.014 

0.988 

Ability  to  Reduce  Problems 

0.150 

0.012 

1.000 
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3.6  Step  6:  Alternative  Generation 

Once  the  weighting  of  the  value  hierarchy  was  complete,  the  next  step  was  to 
generate  the  alternatives.  However,  there  were  seven  contractors  that  competed  for  the 
SABER  contract  at  the  subject  organization.  Therefore,  the  alternatives  were  already 
provided  and  this  step  was  not  necessary.  The  recent  source  selection  conducted  at  the 
subject  organization  provided  a  current  source  of  contractor  alternatives  along  with  their 
corresponding  past  perfonnance  and  technical  information.  The  seven  contractors  were 
identified  as  Contractor  A  through  G  to  ensure  anonymity. 

3.7  Step  7:  Alternative  Scoring 

The  final  step  before  conducting  analysis  is  to  score  the  alternatives.  As  stated  in 
the  previous  step,  empirical  data  was  obtained  through  7  contractors  who  competed  for 
the  recent  award  of  a  SABER  contract.  This  data  was  used  to  score  each  contractor 
against  the  value  functions  created  for  each  measure  within  the  model.  Since  access  to 
coefficient  data  is  restricted,  notional  data  was  randomly  generated  for  each  contractor 
alternative  utilizing  a  random  number  generator.  A  list  of  all  the  scores  generated  for 
each  alternative  contractor  is  included  in  Appendix  F. 
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Chapter  IV.  Results  and  Analysis 


This  chapter  provides  the  results  of  the  deterministic  and  sensitivity  analyses 
conducted  in  Steps  8  and  9  of  the  value  focused  thinking  (VFT)  process.  Step  8, 
deterministic  analysis,  provided  the  results  of  the  model  by  generating  a  rank  ordered  list 
of  the  alternatives.  From  these  results,  the  amount  of  impact  that  the  values  and  measures 
have  on  the  alternative  can  also  be  observed.  Step  9,  sensitively  analysis,  provides 
insight  into  how  an  alternative’s  ranking  can  vary  if  weighting  values  were  to  change. 
This  analysis  gives  the  decision  maker  insight  into  which  values  or  measures  may  be 
sensitive  to  weighting  value  changes,  along  with  the  variation  in  weighting  values  that 
would  cause  changes. 

4.1  Step  8:  Deterministic  Analysis 

The  deterministic  analysis  is  calculated  utilizing  the  “values”  that  are  scored 
against  the  SDVFs  of  the  measures  and  the  corresponding  global  weighting  values,  which 
are  then  incorporated  together  into  the  additive  value  function.  The  end  result  is  a  rank 
ordering  of  the  alternatives  in  which  each  total  value  will  range  between  0  and  1 .  The 
resulting  detenninistic  analysis  of  the  7  contractors  used  in  this  research  is  shown  in 
Figure  12.  The  overall  values  for  the  competing  contractors  range  from  0.578  to  0.874. 

It  is  important  to  note  that  these  values  only  represent  the  ordinal  relationship  between 
the  alternatives;  there  is  no  ratio  relevance  to  be  gained  from  the  values.  For  example, 
the  value  of  Contractor  D  is  0.818  and  the  value  of  Contractor  B  is  0.778.  This  does  not 
mean  that  Contractor  D  is  0.04  value  point  or  5.1%  better  than  Contractor  B;  the  values 
simply  signify  that  Contractor  D  appears  to  be  the  better  of  the  two  alternatives. 
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Contractor  E 
Contractor  A 
Contractor  D 
Contractor  B 

Contractor  F 
Contractor  G 
Contractor  C 


□  Coefficient 

□  Primary  or  Subcontractor 

□  In-house  Subcontract  out 

□  Rehire 

■  Cooperative 

□  Contracting  Actions  Against 

□  Scope  Multidiscipline 

□  Size  $  Amount 

■  Military  Instillation  Experience 

□  Met  Performance  Standards 

□  Submittals  in  Timely  Manner 

□  Resolve  Delays 

■  Completion  of  Punch-list  Items  ■  Warranty  in  Timely  Manner 

□  Compliance  Laws  and  Regs 

□  Safety  Plan 

□  Contract  Requirements 

□  Ability  to  Reduce  Problems 

□  On-site  Presence 

□  Quality  Control  Plan 

□  Quality  Workmanship 

□  Materials  Used  Specified 

□  Adequate  Submittals 

□  Compliance  Regs  &  Code 

Figure  12.  Overall  Ranking  for  Competing  Contractors 


There  is  a  tremendous  amount  of  insight  and  information  that  can  be  gathered  from 
deterministic  analysis.  For  example,  Figure  12  shows  that  Contractor  E  represents  the 
best  alternative.  However,  if  the  Lowest  Price  Technically  Acceptable  (LPTA)  method 
of  the  current  SABER  source  selection  process  were  used,  the  contractors  would  be 
evaluated  in  rank  order  of  their  pricing  score  as  shown  in  Figure  13.  The  first  contractor 
that  would  be  evaluated  under  the  LPTA  system  would  be  Contractor  D.  When  all  values 
are  considered  though,  Contractor  E  represents  the  best  choice;  yet  under  the  LPTA 
source  selection  process,  Contractor  E  is  in  third  place  and  may  not  even  get  evaluated. 
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Not  only  is  Contractor  E  a  strong  competitor,  but  Contractor  A  is  also  considered  a  more 
desirable  alternative  than  either  Contractor  D  or  B.  Additionally,  Contractor  C  received 
no  value  for  price  in  this  model  because  of  an  extremely  high  coefficient  value. 
However,  an  alternative  could  also  receive  no  value  for  price  due  to  an  extremely  low 
coefficient  value,  yet  would  be  moved  to  the  top  of  the  evaluation  order  using  LPTA. 


Contractor  E 

0.700 

Contractor  A 

0.700 

Contractor  D 

1.000 

Contractor  B 

0.850 

Contractor  F 

0.500 

Contractor  G 

0.400 

Contractor  C 

0.000 

Figure  13.  Contractor  Rankings  against  Price  (Coefficient) 


Deterministic  analysis  can  also  be  used  to  compare  the  contractors  across  all 
values  of  a  particular  tier,  for  a  set  of  values  or  measures  within  a  given  branch,  or  for  a 
value  or  measure  individually.  For  example,  the  first  tier  of  Price,  Past  Performance,  and 
Technical  Capabilities  can  be  presented  as  Figure  14  to  better  illustrate  how  each 
contractor  is  represented  in  each  of  to  the  top  three  values.  The  contractors  were  already 
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broken  down  by  price  in  Figure  13,  so  Figure  15  illustrates  the  contractors  scores 
represented  by  non-pricing  values  only.  From  this  analysis,  insight  can  be  gained  into  the 
particular  strengths  and  weaknesses  of  the  contractors.  For  example,  based  on  non¬ 
pricing  values,  Contractor  C  would  now  be  ranked  third  and  Contractor  F  would  be  in 
fourth  place. 


Contractor  E  0.874 
Contractor  A  0.842 
Contractor  D  0.818 
Contractor  B  0.778 
Contractor  F  0.713 
Contractor  G  0.643 
Contractor  C  0.578 


Price  ■  Past  Performance  □  Technical  Capabilities 


Figure  14.  Overall  Ranking  for  Competing  Contractors  (First  Tier  Values) 
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Contractor  E 

0.959 

Contractor  A 

0.912 

Contractor  D 

0.728 

Contractor  B 

0.742 

Contractor  F 

0.818 

Contractor  G 

0.762 

Contractor  C 

0.863 

■  Past  Performance  □  Technical  Capabilities 


Figure  15.  Contractor  Rankings  against  Past  Performance  and 
Technical  Capabilities 


The  application  of  deterministic  analysis  can  continue  down  the  tier  levels  and 
branches  to  show  as  much  or  as  little  detail  as  the  decision  makers  desire.  If  the  breakout 
of  how  the  second  tier  values  under  Technical  Capabilities  is  desired,  then  Figure  16  is 
generated.  Immediately  it  can  be  noted  that  Quality  Control  contributes  a  significant 
portion  to  the  value  and  that  Labor  Standards  has  a  large  variation  between  the 
contractors.  Conducting  this  type  of  analysis  can  help  pinpoint  how  contractors  may  be 
differing  from  one  another  or  which  contractor  lacks  significance  in  a  highly  weighted 
value.  Contractor  B  and  Contractor  C  have  essentially  the  same  score  for  Technical 
Capabilities,  but  looking  at  analysis  output,  it  is  easily  noted  that  Contractor  B  is  better  at 
“Quality  Control”  and  Contractor  C  is  better  in  “Management  Effectiveness.” 
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Contractor  E 

0.941 

Contractor  A 

0.886 

Contractor  D 

0.625 

Contractor  B 

0.668 

Contractor  F 

0.840 

Contractor  G 

0.667 

Contractor  C 

0.864 

□  Labor  Standards  H  Safety  Standards 

□  Management  Effectiveness  ■  Quality  Control 


Figure  16.  Second  Tier  Values  under  Technical  Capabilities 


Often,  decision  makers  are  concerned  with  an  alternative’s  score  for  highly 
weighted  measures.  The  non-pricing  measures  with  the  highest  global  weights  for  the 
SABER  contractors  are:  “Compliance  Laws  and  Regs,”  “Compliance  Regs  and  Code,” 
“Safety  Plan,”  and  “Contract  Requirement;”  these  are  shown  in  Figure  17.  This  analysis 
shows  that  Contractor  A,  F,  and  C  are  relatively  close  for  a  second  place  overall  rating 
based  on  these  measures.  Additionally,  Contractors  G,  D,  and  B  all  score  poorly,  which 
makes  it  difficult  for  them  to  be  competitive  overall. 
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Contractor  E 

0.939 

Contractor  A 

0.845 

Contractor  D 

0.600 

Contractor  B 

0.648 

Contractor  F 

0.861 

Contractor  G 

0.645 

Contractor  C 

0.848 

□  Compliance  Laws  and  Regs  ■  Compliance  Regs  &  Code 

□  Safety  Plan  ■  Contract  Requirements 


Figure  17.  Contractor  Rankings  for  Highly  Weighted  Measures 


The  deterministic  analyses  allows  the  decision  makers  to  easily  decipher  how  the 
measures  influence  the  alternatives  and  helps  to  sort  through  and  organize  a  normally 
very  subjective  process.  This  insight  can  be  extremely  useful  for  documentation 
purposes  as  well.  If  a  contractor  has  questions  about  how  they  are  ranked  or  where  their 
strength  and  weaknesses  are,  insight  can  be  quickly  and  easily  attained  from  a 
deterministic  analysis  as  shown  in  the  previous  examples. 

Several  different  iterations  of  deterministic  analyses  can  be  executed  to  provide  as 
much  insight  as  a  decision  maker  deems  necessary.  For  instance,  a  common  use  of  the 
deterministic  analysis  is  to  create  hypothetical  alternatives  by  varying  the  values  of  one 
or  two  measures  within  an  existing  alternative.  For  example,  Contractor  C  did  not 
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receive  any  value  for  price  due  to  a  poor  coefficient  value.  What  if  Contractor  C  had  a 
highly  desirable  coefficient  value?  How  would  it  have  ranked  against  the  other 
contractors  if  all  other  measures  were  held  constant?  A  hypothetical  alternative  can  be 
created  to  answer  such  questions,  as  shown  by  Contractor  C(l)  in  Figure  18.  Therefore, 
if  Contractor  C  had  submitted  a  desirable  coefficient  value,  it  would  have  been  ranked 
first. 
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□  Coefficient 

■  Primary  or  Subcontractor 

□  In-house  Subcontract  out 

□  Rehire 

■  Cooperative 

□  Contracting  Actions  Against 

□  Scope  Multidiscipline 

□  Size  $  Amount 

■  Military  Instillation  Experience 

□  Met  Performance  Standards 

□  Submittals  in  Timely  Manner 

□  Resolve  Delays 

■  Completion  of  Punch-list  Items  ■  Warranty  in  Timely  Manner 

□  Compliance  Laws  and  Regs 

□  Safety  Plan 

□  Contract  Requirements 

□  Ability  to  Reduce  Problems 

□  On-site  Presence 

□  Quality  Control  Plan 

□  Quality  Workmanship 

□  Materials  Used  Specified 

□  Adequate  Submittals 

□  Compliance  Regs  &  Code 

Figure  18.  Contractor  Rankings  Including  Hypothetical  Contractor  C(l) 
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4.2  Step  9:  Sensitivity  Analysis 

The  sensitivity  analysis  is  a  unique  and  extremely  insightful  aspect  of  the  VFT 
methodology.  The  purpose  of  the  sensitivity  analysis  is  to  detennine  how  changes  in  the 
model  will  impact  the  overall  decision.  Often,  changes  within  the  weights  assigned  to 
values  and  measures  are  explored  to  pinpoint  which  values  are  sensitive  to  changes,  and 
which  ones  are  not  sensitive  to  changes.  By  exploring  these  variations,  the  decision 
maker  can  gain  insight  into  how  the  alternatives  may  be  influenced  by  changing  certain 
values,  as  well  as  which  values  may  not  be  necessary  or  may  be  extremely  influential. 

Since  price  is  an  area  of  high  concern  in  the  selection  of  a  SABER  contractor,  and 
because  it  possesses  such  a  large  global  weight,  the  first  sensitivity  analysis  was  applied 
to  the  “Price”  value.  This  analysis  will  provide  a  graphical  representation  of  how  the 
contractors  will  vary  in  rank  when  the  weight  assigned  to  the  “Price”  value  is  allowed  to 
range  from  0  to  1  while  holding  the  weights  of  the  other  values  on  the  same  tier,  “Past 
Performance”  and  “Technical  Capabilities,”  proportionally  constant.  As  shown  in  Figure 
19,  this  value  is  slightly  sensitive  to  the  weight  change  due  to  no  change  in  the  top 
alternative  until  a  moderate  increase  in  the  weight  from  0.330  to  0.470  is  reached.  This  is 
also  an  example  of  a  value  that  is  not  sensitive  to  decreasing  weighting  values,  but  is 
slightly  sensitive  to  increasing  weighing  values. 
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Figure  19.  Sensitivity  Graph  for  Price 


This  is  a  very  useful  tool  for  a  SABER  source  selection.  By  looking  at  the  weight 
value  determined  in  the  model  (0.330),  it  is  apparent  that  Contractor  E  is  the  top 
alternative.  However,  an  increase  in  weighted  value  of  “Price”  will  cause  Contractor  D 
to  slope  upward  and  Contractor  E  to  slope  downward  due  to  the  lower  price  of  Contractor 
D.  At  approximately  0.470  it  can  be  seen  that  Contractors  E  and  D  crisscross;  for  that 
weight  value  and  above,  Contractor  D  is  the  top  alternative. 
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Figure  19  shows  Contractor  D  as  the  top  value  only  after  the  weight  is  increased  to 
0.470  and  above.  The  SABER  RFP  (2004)  states:  “For  those  proposals  detennined  to  be 
acceptable,  a  tradeoff  between  price  and  past  performance  will  be  conducted  with  past 
performance  being  significantly  more  important  than  price.”  Since  past  performance 
(non-pricing  factors)  should  be  weighted  significantly  more  than  price,  it  does  not  make 
sense  to  consider  what  will  happen  in  the  weighted  range  of  0.47  and  above  on  price. 

The  LPT  A  process  elects  to  evaluate  the  low  priced  contractor  as  a  first  priority,  which  is 
the  same  as  moving  to  the  far  right  (weighting  value  of  1 .0)  on  the  sensitivity  graph. 

Other  indicators  of  sensitivity  are  shallow  or  steeply  sloping  lines  of  alternatives. 
The  more  sensitive  an  alternative  is  to  a  given  value  or  measure,  the  steeper  its  sloping 
line  will  appear  due  to  a  greater  value  change  through  very  little  change  in  weighting. 
Contractor  C  from  Figure  19  is  an  example  of  an  alternative  showing  extreme  sensitivity 
to  a  given  value.  This  coincides  with  the  deterministic  analysis,  as  Contractor  C  did  not 
receive  any  value  from  its  score  for  the  measure  of  “Price.”  This  analysis  can  help 
pinpoint  areas  that  may  need  further  examination  or  explanation. 

Past  Perfonnance  is  another  value  with  a  large  influence  on  the  SABER 
contractor  decision  and  has  a  sensitivity  analysis  shown  in  Figure  20.  This  is  an  example 
of  an  insensitive  value  because  there  is  no  change  in  the  top  alterative  across  the  entire 
range  of  weighting  values.  Another  indicator  that  this  value  is  insensitive  is  the  shallow 
sloping  lines  of  several  alternatives. 
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- Contractor  A 

—  -  -  Contractor  B 

—  -  -  Contractor  C 
—  Contractor  D 

Contractor  E 
- -  Contractor  F 

—  -  Contractor  G 


Figure  20.  Sensitivity  Graph  for  Past  Performance 


Figure  21  shows  a  sensitivity  analysis  for  that  last  highly  (0.300  or  above  global 
weight)  weighted  value  within  the  model.  This  analysis  shows  an  example  of  a  value  that 
is  insensitive  to  increasing  weighting  values,  but  is  slightly  sensitive  to  decreasing 
weighting  values.  The  top  alternative  would  change  if  the  weighting  value  were 
decreased  to  approximately  0.185  or  lower. 
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Figure  2 1 .  Sensitivity  Graph  for  Technical  Capabilities 


Sensitivity  analysis  can  provide  insight  into  measures  that  may  not  have  an 
overall  large  global  weight,  but  still  may  play  a  significant  role  in  differentiating  between 
two  highly  competitive  contractors.  If  such  an  instance  were  to  arise,  the  sensitivity 
analysis  of  that  measure  could  help  determine  how  practical  it  would  be  for  one 
contractor  to  outscore  another.  Figure  22  shows  the  sensitivity  graph  for  the  measure 
“Quality  Control  Plan.”  If  the  weighting  value  increased  to  approximately  0.100  or 
greater,  Contractor  A  would  overtake  Contractor  E  as  the  better  alternative.  However, 
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with  a  closer  look  at  the  model,  the  measure  “Quality  Control  Plan”  along  with  four  other 
measures  fall  under  the  value  “Quality  Control.”  The  global  weight  of  “Quality  Control” 
is  only  0.132;  therefore,  the  chance  of  weighting  the  “Quality  Control  Plan”  measure 
0.100  or  higher  and  assigning  the  remaining  weighting  (0.032  or  less)  to  the  other  four 
measures  is  small.  The  impracticality  of  such  a  scenario  is  even  more  evident  knowing 
the  original  weighting  value  of  “Quality  Control  Plan”  is  0.020.  A  500%  increase  in 
assigned  weighting  for  the  measure  would  have  to  take  place. 


Figure  22.  Sensitivity  for  Quality  Control  Plan  Measure 
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Chapter  V.  Conclusions  and  Recommendations 


This  chapter  provides  the  closing  answers  to  the  research  and  investigative 
questions  from  Chapter  1,  a  summary  of  the  final  questionnaire  answers,  and  a  summary 
of  the  results  and  potential  future  efforts  relating  to  this  research.  The  results  presented  in 
this  chapter  correspond  to  Step  10  of  the  Value  Focused  Thinking  (VFT)  process. 

Closing  comments  will  be  presented  on  the  potential  incorporation  of  a  VFT  model  in 
Simplified  Acquisition  of  Base  Engineer  Requirements  (SABER)  source  selections  along 
with  strengths  and  weakness  observed  in  this  research  effort.  Lastly,  recommendations 
for  future  research  relating  to  this  topic  will  be  synopsized. 

5,1  Process  Overview 

The  motivation  for  this  research  was  to  apply  an  objective  analysis  tool  to  the 
subjective  analysis  process  of  a  SABER  source  selection.  In  doing  so,  decision  makers 
and  subject  matter  experts  with  recent  exposure  to  a  SABER  source  selection  were 
consulted.  The  VFT  model  was  constructed  to  illustrate  the  possible  benefits  and  insights 
that  can  be  gained  by  incorporating  a  decision  making  tool  within  the  evaluation  process. 
The  model  was  developed  using  the  VFT  10-step  method  (Shoviak,  2001);  pre-  and  post¬ 
questionnaires  were  used  to  gain  further  insight  into  answering  the  research  question 
posed  in  Chapter  1 :  What  are  all  the  criteria,  values,  and  measures  detennined  essential 
by  the  decision  makers  to  effectively  select  the  best  contractor  and  why  is  it  essential  to 
evaluate  all  the  alternatives?  To  help  fully  answer  the  research  question,  the  following 
investigative  questions  were  developed  and  answered  during  the  course  of  this  research: 
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1)  How  well  does  the  current  selection  process  work? 

2)  What  are  the  criteria,  to  include  price  and  past  performance,  that  are  deemed 
necessary  to  evaluate  competing  contractors  and  what  are  the  underlying 
measures  for  these  criteria? 

3)  How  are  the  measures  weighted  by  the  decision  makers? 

4)  How  well  does  the  VFT  model  convey  the  true  felling  of  the  decision  makers? 

5)  How  does  the  VFT  model  compare  to  the  current  selection  process? 

The  first  investigative  question  was  answered  with  the  literature  research  discussed 
in  Chapter  2  and  the  initial  questionnaire  discussed  within  Chapter  3.  The  second  and 
third  questions  were  answered  in  steps  one  through  five  of  the  VFT  process.  The 
hierarchy  was  comprised  of  first  tier  values  of  Price,  Past  Performance,  and  Technical 
Capabilities.  These  values  were  further  broken  down  into  7  second-tier  values,  2  third- 
tier  values,  and  24  measures.  Value  functions  were  developed  and  defined  for  the 
measures  to  objectively  determine  the  level  of  attainment  each  contractor  represented, 
and  local  and  global  weightings  were  applied  to  the  values  and  measures  throughout  the 
hierarchy. 

The  weighting  of  the  value  hierarchy  shows  that  Price  is  a  significant  factor  in  the 
selection  of  a  contractor  which  is  to  be  expected.  The  “Coefficient”  measure  has  the 
biggest  influence  in  the  evaluation  process  (33%)  and  can  make  or  break  the  contractor’s 
competitive  probability.  The  Past  Performance  and  Technical  Capabilities  values  along 
with  encompassing  measures  make  up  the  other  67%  of  the  decision  influence,  which 
holds  to  the  intent  of  having  non-price  related  factors  significantly  outweighing  price 
factors  as  stated  in  the  RFP.  More  importantly,  these  assigned  weightings  hold  constant 
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through  the  evaluation  process,  which  cannot  conclusively  be  defended  by  the  current 
SABER  source  selection  process. 

The  fourth  and  fifth  questions  were  answered  by  completing  steps  six  through  nine 
of  the  VFT  process.  This  was  accomplished  by  utilized  the  past  performance  information 
of  seven  competing  contractors  to  generate  the  alternatives  to  be  evaluated.  The 
alternatives  were  then  scored  accordingly  against  the  value  functions  and  deterministic 
and  sensitivity  analysis  was  conducted. 


5.2  VFT  Model  -  Current  SABER  Source  Selection  Comparison 

Additional  insight  into  the  fourth  and  fifth  investigative  questions  as  well  as  the 
overall  research  question  was  obtained  through  a  final  questionnaire  (Appendix  G) 
completed  by  the  research  subjects.  The  following  are  the  collective  concluding  thoughts 
gained  from  the  questionnaire: 

•  VFT  process  makes  it  easier  to  see  how  values/measures  influence  the 
evaluation  process  and  provides  a  clear  understanding  of  all  evaluation 
criteria. 

•  Explaining  the  rational  of  a  decision  under  the  current  process  is 
extremely  difficult  without  quantifying  numbers  -  VFT  provides 
quantifying  results  and  clearly  defines  not  only  the  best  choice  contractor, 
but  how  contractors  compare  to  one  another. 

•  VFT  process  can  potentially  reduce  need  for  re-evaluations.  One 
evaluation  deemed  realistic  possibility  due  to  details  provided  in 
deterministic  and  sensitivity  analysis. 

•  VFT  process  is  a  great  technique  for  providing  accurate  and  complete 
documentation  of  the  evaluation  process. 

•  VFT  is  viewed  as  a  necessary  tool  for  SABER  source  selections  and  is 
recommended  for  implementation. 
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From  these  comments  and  discussions  with  the  research  subjects,  the  VFT  process  was 
viewed  as  a  great  tool  to  help  alleviate  the  problems  that  arise  in  the  current  subjective 
analysis  process.  The  subjects  liked  being  able  to  evaluate  all  competing  contractors, 
being  involved  in  all  evaluation  criteria  (especially  price),  and  the  objective  qualities  and 
visual  graphs  provided  by  the  deterministic  and  sensitivity  analysis.  These  attributes  are 
not  available  under  the  current  SABER  source  selection  process. 

5.3  Conclusions 

The  main  conclusion  provided  by  this  thesis  is  concise:  VFT  is  a  viable  and 
essential  tool  for  the  SABER  source  selection  process.  There  are  several  problems  with 
the  current  source  selection  process,  to  include:  there  is  no  way  to  hold  intended 
weighting  constant  (i.e.,  non-pricing  factors  significantly  more  important  than  pricing 
factors)  throughout  the  evaluations,  all  competing  contractors  are  not  evaluated,  lowest 
priced  contractor  is  evaluated  first  which  introduces  immediate  evaluator  bias,  there  is  no 
way  to  objectively  differentiate  between  contractors,  and  all  source  selection  committee 
members  are  not  allowed  to  provide  input  on  all  evaluation  factors  (i.e.,  price). 

VFT  can  remedy  these  problems  through  the  objective  analysis  process  described 
within  this  thesis.  More  importantly,  realistic  data  was  utilized  to  evaluate  seven 
competing  contractors  and  showed  the  importance  of  evaluating  all  contractors,  holding 
weighting  constant,  having  an  ability  to  evaluate  all  criteria  (measures),  and  having  an 
ability  to  compare  and  contrast  competing  contractors  through  implementation  of  the 
model  with  the  use  of  deterministic  and  sensitivity  analysis.  The  value  hierarchy  and 
value  functions  could  also  be  utilized  to  help  specify  the  evaluation  factors  more  clearly 
within  the  SABER  Request  for  Proposal  (RFP).  The  final  decision  is  still  the 
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responsibility  of  the  Source  Selection  Authority  (SSA),  but  a  VFT  tool  which  can  aid  in 
the  probability  of  selecting  the  best  value  SABER  contractors  for  the  Air  Force  can  also 
aid  and  increase  the  success  and  potential  of  SABER  programs. 

5.4  Model  Strengths 

This  model  provides  an  objective  and  systematic  development  in  which  all 
evaluation  criteria  and  all  competing  contractors  can  be  synthesized  into  a  coherent 
evaluation  process.  The  data  gathered  for  evaluation  purposes  is  easily  transformed  into 
quantifiable  data  which  makes  it  easier  to  differentiate  and  analyze  the  alternatives. 
Additionally,  the  model  enables  all  alternatives  considered  to  be  evaluated  in  a  systematic 
fashion  to  eliminate  bias  influences.  Instead  of  relying  on  subjective  judgmental 
comments,  this  model  requires  complete  and  objective  input  which  is  documented  and 
defendable.  Analysis  can  be  conducted  on  any  value,  measure,  or  weighting  situation 
within  the  model,  making  it  extremely  versatile  and  insightful  to  decision  makers  and 
competing  contractors  alike. 

The  VFT  methodology  caters  to  the  values  and  preferential  weightings  of  the 
decision  makers  who  utilize  the  model.  These  values  are  what  the  Air  Force  is  concerned 
with  when  selecting  a  SABER  contractor,  and  a  VFT  model  ensures  that  all  values  are 
incorporated  and  evident  throughout  the  evaluation  process.  The  model  is  also  adaptable 
to  the  values  and  weightings  derived.  This  enables  pertinent  addition  or  deletion  of 
values  and  measures;  changes  in  weighting  to  reflect  the  views  of  different  decision 
makers  are  also  possible. 
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5.5  Model  Limitations 


The  main  limitation  of  this  model  stems  from  the  development  of  data  obtained  at  a 
sole  location.  In  utilizing  one  Air  Force  base,  data  was  obtained  exclusively  from  one 
source  of  subject  matter  experts.  Additional  infonnation  and  data  input  for  value 
hierarchy  construction  and  weighting  values  could  have  been  enhanced  by  incorporating 
input  from  several  Air  Force  bases.  Therefore,  the  current  model  may  have  some 
inherent  bias  geared  to  the  needs  of  the  location  investigated.  The  intent  of  this  research, 
however,  was  an  initial  attempt  to  illustrate  the  usefulness  of  a  VFT  model  in  a  SABER 
source  selection  process;  therefore,  a  sole  location  with  real-world  data  and  the  necessary 
availability  of  research  subjects  provided  the  best  scenario. 

Another  limitation  to  this  model  is  the  incorporation  of  price  within  the  value 
hierarchy.  The  source  selection  committee  collectively  wants  to  evaluate  price;  however, 
including  price  in  the  model  was  extremely  difficult  for  two  reasons.  First,  there  is  only 
one  measure  to  evaluate  price  for  a  SABER  contractor  and  that  is  through  the 
“Coefficient”  measure,  which  represented  33%  of  the  global  weight  of  the  model. 

Second,  the  coefficient  value  range  is  extremely  influential  and  difficult  to  pinpoint.  The 
decision  makers  do  not  want  too  low  of  a  coefficient,  which  will  potentially  cause  a 
contractor  to  be  unsuccessful  throughout  the  course  of  the  contract.  However,  they  also 
do  not  want  a  coefficient  that  is  too  high,  potentially  gouging  the  resources  of  the 
Government.  The  value  range  determined  acceptable  for  this  model  is  notional  and  was 
obtained  from  the  subject  matter  experts.  The  construction  of  the  value  function  for  this 
measure  is  in  need  of  refinement,  but  served  to  illustrate  the  application  and  usefulness  of 
the  model  as  a  whole. 
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5.6  Future  Research 


This  research  was  an  initial  attempt  to  gain  insight  into  the  problem  areas 
associated  with  the  SABER  source  selection  process  and  provide  an  avenue  to  help 
alleviate  those  problem  areas;  therefore,  there  are  several  areas  for  future  research  work. 
One  main  area  is  to  broaden  the  participation  from  one  Air  Force  base  to  several  bases. 
This  would  enable  a  wider  variety  of  input  from  subject  matter  experts  which  could 
possibly  result  in  changes  to  the  value  hierarchy  constructed  in  this  thesis  or  provide 
added  validity  to  the  values  and  measures  within  the  model.  Additionally,  a  universal 
value  hierarchy  and  weightings  may  be  developed  by  researching  several  bases. 

Another  area  for  research  is  to  incorporate  members  from  the  contracting  squadrons 
on  a  larger  scale  and  incorporate  their  thoughts  and  ideas  within  the  model.  Doing  so 
will  help  validate  the  model  as  a  source  selection  tool.  Additionally,  contracting 
personnel  can  help  solve  the  problems  that  were  discussed  with  the  “Coefficient” 
measure. 

Lastly,  additional  research  could  be  conducted  by  incorporating  the  VFT  model 
within  an  actual  SABER  source  selection  to  analyze  how  effective  and  efficient  it  would 
be  to  the  source  selection  committee  in  a  real-time  situation.  Additional  insight  into  the 
model,  the  source  selection  process,  and  the  source  selection  committee  along  with  areas 
for  improvement  would  be  gained  from  this  process. 
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Appendix  A.  Past  Performance  Grading  Sheets 


F22608-04-R-0002 
Attachment  9 

PAST  AND  PRESENT  PERFORM  ANCE  QUESTIONNAIRE. 

WHEN  FILLED  IN  THIS  DOCUMENT  IS  SOURCE  SELECTION  SENSITIVE  INFORMATION  IN 
ACCORDANCE  WITH  FAR  3.104 

PART  I:  OFFEROR  COMPLETE  BELOW: 

1 .  Contractor  (Name,  Address,  and  Zip  code): 


2.  Contract  Number:  _ 

3.  Program/Project  Title: _ 

4.  Type  of  Contract:  Fixed  Price  _  Cost  Reimbursement _  Other  _  (Explain) 


5.  Description,  location,  and  relevancy  of  work. 


6.  Performance  Period: _ 

7.  Date  of  Award: _ _ 

8.  Original  Contract  Dollar  Value: _ 

9.  Current/Completed  Contract  Dollar  Value:  _ 

10.  If  amounts  for  8  and  9  above  are  different,  provide  a  brief  explanation  for  the  reason. 


1 1 .  Contract  Completion  Date  (including  extension): 

12.  Point  of  Contact  Information: 

Procuring/ Administrative  Contracting  Officer: 

Name: _ Phone :_ 

Address: _ Fax: _ 

_ e-mail 


Page  1  of  6 
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F22608-04-R-0002 
Attachment  9 


PART  11.  RESPONDENT/GOVERNMENT  OFFICIAL’S  INFORMATION  AND  PERFORMANCE 
EVALUATION 

You  have  been  provided  as  a  reference  in  response  to  the  solicitation  for  a  contract  to  provide  for  basewide  paving 
IDIQ.  In  order  for  the  government  to  evaluate  the  past  performance  of  the  firm  providing  you  as  a  reference,  please 
complete  this  questionnaire  and  forward  to  the  14'h  Contracting  Squadron  in  accordance  with  the  instructions  listed 
below. 

The  individual  knowledgeable  of  the  contractor’s  day-to-day  operations  and  overall  performance  should  complete 
this  questionnaire.  However,  that  individual  is  encouraged  to  supplement  their  own  knowledge  of  the  contractor’s 
performance  with  the  judgement  of  others  within  their  organization,  as  applicable. 

Offer  Due  Date  and  Time: _ 

FAX  to:  (662)  434-3049  or  DSN  742-3049 

E-MAIL  to:  Ramona.allen@columbus.af.mil 

Mail  to:  14th  Contracting  Squadron/LGCA 
Atm:  Ramona  Allen 
555  Seventh  Street,  Suite  113 
Columbus  AFB,  MS  39710-1006 

POC:  Ramona  Allen,  Contract  Specialist,  662-434-7785  or  DSN  742-7785 

Or  Waimon  Hendrix,  Contracting  Officer,  662-434-7796,  DSN  742-7796 

1 .  Respondcnt/Govemment  Official’s  Information 

Name:  _ 

Position/Title: _ _ 

Address:  _ _ 


Telephone  Number:  _ Fax: 

E-mail:  _ 


2.  Relationship  and  time  involved  with  Contractor: 


3.  Date  questionnaire  was  completed: 
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Attachment  9 


The  following  chart  depicts  the  RATINGS  that  are  to  be  used  to  evaluate  the  contractor's  performance: 


u 

M 

N 

s 

G 

E 

Unsatisfactory/No 

Confidence 

Marginal/Little 

Confidence 

Neutral/Unknown/ 

Confidence 

Satisfactory/ 

Confidence 

Very  Good/Significan 
Confidence 

Exceptional/High 

Confidence 

Based  on  the  offeror's 
performance  record, 
extreme  doubt  exists 
that  the  successful 
offeror  will  perform 
the  required  effort. 

Based  on  the 
offeror’s 
performance 
record,  substantial 
doubt  exists  that 
the  successful 
offeror  will 
perform  the 
required  effort. 

No  performance 

record 

identifiable. 

Based  on  the 
offeror's 

performance  record, 
some  doubt  exists 
that  the  successful 
offeror  will  perform 
the  required  effort. 

Based  on  the  offeror's 

performance  record, 
little  doubt  exists  that 
the  successful  offeror 
will  perform  the 
required  effort 

Based  on  the  offeror's 

performance  record, 
essentially  no  doubt 
exists  that  the  successful 
offeror  will  successfully 
perform  the  required 
effort. 

For  any  neutral,  marginal,  or  unsatisfactory  rating,  please  provide  explanatory  narratives  in  the  remarks  block. 
These  narratives  need  not  be  lengthy,  just  detailed.  If  a  question  is  not  applicable,  mark  N/A.  If  more  space  is 
needed,  use  the  back  of  the  questionnaire  or  attach  additional  pages.  Be  sure  to  identify  your  continued  narration 
with  the  respective  line  number. 

CONTRACT  WRITTEN  BY: _ 

CONTRACT  NUMBER:  _ 

POINT  OF  CONTACT:  _ 


NAME  OF  PRIME  CONTRACTOR: 


1. 

OUALITY  CONTROL: 

CIRCLE 

ONE 

a. 

Provided  quality  of  workmanship 

u 

M 

N 

S 

G 

E 

b. 

Provided  and  followed  effective  Quality  Control,  (QC)  Plan 

u 

M 

N 

s 

G 

E 

c.  Provided  effective  inspection  procedures  to  meet  contract 
requirements 

u 

M 

N 

s 

G 

E 

d. 

Stored  materials  properly 

u 

M 

N 

s 

G 

E 

e. 

Used  materials  specified 

u 

M 

N 

s 

G 

E 

f. 

Ordered  and  used  adequate  materials 

u 

M 

N 

s 

G 

E 

%■ 

Provided  adequate  submittals 

u 

M 

N 

s 

G 

E 

h. 

Provided  adequate  as-built  drawings 

u 

M 

N 

s 

G 

E 

i. 

Identified/corrected  deficient  work  in  a  timely  manner 

u 

M 

N 

s 

G 

E 

j- 

Suggested  solutions  and  displayed  initiative  to  implement  solutions 

u 

M 

N 

s 

G 

E 

ADDITIONAL  COMMENTS: 
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Attachment  9 

2.  TIMELY  PERFORMANCE: 

CIRCLE 

ONE 

a.  Provided  submittals  in  a  timely  manner 

U 

M 

N 

S 

G 

E 

b.  Researched  submittals  well  and  clearly  identified  the  proposed  item 

U 

M 

N 

s 

G 

E 

c.  Matched  submittals  accurately  to  contract  requirement 

U 

M 

N 

s 

G 

E 

d.  Met  established  allotted  work  schedule 

U 

M 

N 

s 

G 

E 

e.  Resolved  delays 

U 

M 

N 

s 

G 

E 

f.  Completed  punchlist  items  timely 

U 

M 

N 

s 

G 

E 

g.  Completed  punchlist  items  adequately 

U 

M 

N 

s 

G 

E 

h.  Completed  work  within  performance  period 

U 

M 

N 

s 

G 

E 

i.  Developed  realistic  progress  schedules 

U 

M 

N 

s 

G 

E 

ADDITIONAL  COMMENTS: 

3.  MANAGEMENT  EFFECTIVENESS: 

a.  General  Business  Practices: 

CIRCLE 

ONE 

1.  Provided  experienced  managers,  and  supervisors  with  the 
technical  and  administrative  abilities  needed  to  meet  contract 
requirements 

U 

M 

N 

s 

G 

E 

2a.  Hired  experienced  subcontractors 

U 

M 

N 

s 

G 

E 

2b.  Paid  subcontractors/suppliers  in  a  timely  manner 

U 

M 

N 

s 

G 

E 

3.  Provided  well  substantiated  pricing  data  for  modifications 

U 

M 

N 

s 

G 

E 

4.  Completed  all  contract  work  prior  to  requesting  final  inspection. 

Left  no  major  discrepancies 

U 

M 

N 

s 

G 

E 

5.  Provided  timely  and  satisfactory  response  to  warranty  issues  after 

U 

M 

N 

s 

G 

E 

completion  of  project 

6.  Cooperated  to  resolve  problems,  attending  meetings,  (as  applicable) 
and  maintained  communication  to  assure  satisfactory  resolution 

U 

M 

N 

s 

G 

E 

7.  Provided  timely  resolution  of  contract  discrepancies 

u 

M 

N 

s 

G 

E 

8.  Obtained  consent  of  surety  for  increases  in  bonding  as  work  in 
progress  increased 

u 

M 

N 

s 

G 

E 
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b.  Pricing: 

1 .  Submitted  proposals  that  accurately  represent  the  work  required  U  M  N 

2.  Provided  supporting  price/cost  information  for  modifications  dial  U  M  N 

were  accurate,  complete  and  reasonable,  (not  over  inflated  or  under 

estimated) 

3.  The  contractor's  overall  pricing  performance  U  M  N 

c.  Site  Manapement/Sunerintendent: 


1 .  Site  Manager  ensured  compliance  with  contract  requirements  and  U 
safety 


2.  Site  manager  acted  promptly  to  resolve  problems  and  ensure 
work  quality 


M  N 


U  M  N 


3.  Delegated  sufficient  authority  to  Site  Manager  to  make  decisions  or  U  M  N 

take  actions  during  project  performance 

4.  Site  manager  consistently  present  on  site  when  work  was  performed  U  M  N 

d.  Contractor’s  overall  Management  Effectiveness  U  M  N 

ADDITIONAL  COMMENTS:  _ 


G 

G 


G 

G 


4.  COMPLIANCE  WITH  LABOR  STANDARDS: 

a.  Submitted  payrolls  in  a  timely  manner 

b.  Completed  payrolls  correctly 


U 

U 


CIRCLE  ONE 
M  N  S  G 
M  N  S  G 


c  Complied  with  labor  laws  and  regulations  with  specific  attention  to  U  M  N  S 
the  Davis-Bacon  Act 


ADDITIONAL  COMMENTS: 


5.  COMPLIANCE  WITH  SAFETY  STANDARDS: 

CIRCLE  ONE 

a.  Provided  adequate  safety  plan 

U 

M  N 

S  G 

E 

b.  Implemented  safety  plan 

U 

M  N 

S  G 

E 

6.  OTHER  PAST  PERFORMANCE  CONSIDERATIONS: 

CIRCLE  ONE 

a  Assessment  of  liquidated  damages  (frequency,  circumstances,  severity 
of  problem  etc.) 

(Yes 

No) 
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b.  Based  on  the  contractor's  overall  performance,  would  this  contractor  be  rc-hired?  If  not, 
explain  in  “Additional  Comments” 

c.  Was  the  contractor  cooperative  in  negotiating  modifications  so  that  they  would  result  in 
fair  and  reasonable  cost  adjustments  to  the  contract? 

d.  The  contractor  provided  all  deliverables  (including  documentation)  in  a  timely  and 
accurate  manner 

e.  Have  there  been  any  labor  disputes  or  violations  of  public  law? 

f.  Have  there  been  any  indications  that  the  contractor  has  had  any  financial  difficulties? 

g.  Contractor  submitted  modifications  and  other  required  proposals  in  a  timely  manner. 

h.  Was  the  contractor  ever  issued  a  Cure  or  Show  Cause  Notice  under  the  referenced 
contract?  If  yes,  explain  outcome  in  “Additional  Comments” 

ADDITIONAL  COMMENTS:  _ 


(Yes 

No) 

(Yes 

No) 

(Yes 

No) 

(Yes 

No) 

(Yes 

No) 

(Yes 

No) 

(Yes 

No) 

7,  Overall  Rating 

a.  How  would  you  rate  the  contractors  performance  U  M  N  S  G  E 

on  an  overall  basis 

ADDITIONAL  COMMENTS:  _ 


Completed  questionnaires  must  be  received  in  1 4:l1  Contracting  Office.  Columbus  AFB  MS  NLT  Offer  Due  Date. 

If  you  have  any  questions,  please  contact _ ,  phone _ . 

THANK  YOU  FOR  YOUR  TIME  AND  QUICK  RESPONSE. 


Date:  1 8  Jun  03 
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Appendix  B.  Evaluation  Sheet 


SOURCE  SELECTION  INFORMATION  -  SEE  FAR  3.104 
FOR  OFFICIAL  USE  ONLY 

EVALUATOR  WORKSHEET 

EVALUATOR:  _ 

CONTRACTOR:  _ 

RATING: 

(  )  Exceptional  (  )  Very  Good  (  )  Satisfactory  (  )  None  (  )  Marginal  (  )  Unsatisfactory 


STRENGTHS: 


SIGNATURE  (EVALUATOR)  DATE 


SOURCE  SELECTION  INFORMATION  -  SEE  FAR  3.104 
FOR  OFFICIAI .  I ISF.  ONI  Y 
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Appendix  C.  Initial  Questionnaire 


Question  1 .  How  is  the  current  evaluation  process  of  SABER  Source  Selection 
conducted? 


Answers: 

Subject  A: 

1. )  Given  first  candidate  and  they  are  rated  based  on  capabilities,  historical  projects, 
evaluations  from  other  military  units  and  evaluation  from  outside  agencies. 

2. )  Rate  each  candidate  until  you  either  give  the  highest  rating  or  you  go  through  all 
candidates.  If  you  hit  the  highest  rating,  you  stop. 


Subject  B: 

Past  performance,  capability,  management,  firm  breadth  of  talent.  The  process  of 
SABER  Source  Selection  is  based  on  a  number  of  factors  to  include  statements  from  base 
contracting  officials  and  engineering  and  contract  management  personnel. 


Subject  C: 

Contracting  receives  bid  packages  to  include  past  performance.  Contracting  establishes 
Tech  Eval  Committee.  Committee  ranks  all  proposals.  Contracting  ranks  all  proposals 
based  upon  cost  factors.  Rank  proposals  based  upon  tech  eval  and  cost. 


Subject  D: 

CE  selects  team  members,  normally  two  or  more.  Each  member  is  provided  background 
and  perfonnance  criteria  for  each  proposal.  These  are  graded  with  strengths  and 
weaknesses  and  ranked  in  priority  order.  Cost  is  not  factor  evaluated  by  CE  team. 
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Question  2:  What  are  the  evaluation  criteria  utilized  in  the  current  SABER  Source 
Selection? 


Answers: 

Subject  A: 

Capabilities  listing,  project  completed,  evaluations  from  contracting  &  CE  at  other  units, 
evaluations  from  outside  agencies. 


Subject  B: 

Past  performance,  price,  and  written  evaluations  from  base  officials. 


Subject  C: 

Experience  involved  in  execution  of  work.  Experience  in  performance  of  like  work. 
Dollar  amount  of  like  work  and  period  work  was  performed.  Technical  capabilities  of 
company  and  production  capabilities. 


Subject  D: 

Past  performance,  experience  or  expertise,  subcontracting,  costs. 
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Question  3:  Is  there  a  clear  understanding  of  all  evaluation  criteria  utilized  in  the  current 
SABER  Source  Selection? 


Answers: 

Subject  A: 

No.  Depending  on  the  evaluator,  one  may  place  higher  marks  on  certain  areas  than 
another  evaluator.  This  can  have  a  dramatic  affect  if,  per  say,  one  person  rates  paper 
work  over  quality  of  work. 


Subject  B: 

Yes.  However,  if  the  information  is  skewed  or  unreliable  the  outcome  of  selecting  a 
SABER  entity  would  be  flawed. 

Subject  C: 

I  feel  that  there  is  not  a  clear  understanding  of  the  criteria  or  the  value  of  each  element  in 
the  selection  process.  I  feel  too  much  justification  on  cost  vs.  past  performance  trade-off 
is  required.  [Per  clarification  with  subject  -  currently  too  much  emphasis  on  cost  rather 
than  past  performance.] 


Subject  D: 
Yes. 
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Question  4:  How  many  times  are  competing  contractors  evaluated  or  re-evaluated? 

Answers: 

Subject  A: 

Maybe  once  but  it  could  go  as  far  as  three  or  four  times  to  break  dead-locks. 

Subject  B: 

Usually,  once.  There  should  be  no  need  for  re-evaluation  unless  two  or  more  contractors 
are  very  close  in  capability  and  technical  expertise. 

Subject  C: 

Twice,  if  there  is  not  a  clear  ranking  of  the  competing  contractors. 

Subject  D: 

Most  are  intended  to  evaluate  once,  but  clarifications  or  changes  prompt  a  re-evaluation 
and/or  proposal  change. 
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Question  5:  If  evaluations  are  done  more  than  once,  what  type  of  information  becomes 
more  evident  by  conducting  re-evaluations? 


Answers: 

Subject  A: 

Comparisons  to  past  history.  If  there  were  completion  issues  or  problems. 

Subject  B: 

Ability  to  execute  -  multi-discipline,  multi-task  jobs  /  subcontractors  affiliated  with  the 
prime  contractor. 

Subject  C: 

Relevance  of  experience  of  past  work  and  customer  satisfaction. 

Subject  D: 

Knowledge  of  the  contract  requirements  may  be  lacking.  Probably  a  contractor  to  avoid. 
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Question  6:  If  an  evaluation  of  a  contractor  was  done  only  once  with  the  current 

evaluation  process,  would  there  be  enough  information  to  conduct  proper 
documentation? 


Answers: 

Subject  A: 

No.  Case  in  point,  we  rated  a  contractor  high  but  we  were  not  shown  his  coefficient.  It 
turned  out  to  be  too  low  and  therefore  we  have  had  to  battle  the  contractor  on  trying  to 
increase  his  margin  on  contracts  trying  to  make  up  the  difference. 


Subject  B: 

Yes.  Provided  all  data  presented  is  accurate  and  provides  an  honest  look  at  the 
contractor. 


Subject  C: 

Not  100%  of  the  time. 

Subject  D: 

Yes.  Most  all  solicitations  provide  a  clause  or  paragraph  to  say  award  may  be  made  with 
no  discussion. 
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Question  7:  Is  there  a  need  for  improvement  in  the  evaluation  process  of  SABER  Source 
Selections? 


Answers: 

Subject  A: 

Yes,  see  answer  for  #6.  Need  to  have  data  fde  on  evaluations  of  each  delivery  order  done 
by  a  contractor  for  the  military.  Utilize  ACES  (mandatory).  In  areas  where 
subcontractors  are  hard  to  find,  eliminate  firms  who  are  90%  contracted  out  to  subs. 


Subject  B: 

Yes.  Too  many  times  the  process  of  source  selection  is  not  conducted  by  person  with  the 
necessary  technical  capability.  People  also  have  a  tendency  to  allow  their  personal  views 
to  get  involved  in  the  process.  There  should  be  a  hard  look  at  the  proposed  contractor  - 
an  ability  to  complete  work,  subs,  and  in-house  capability. 


Subject  C: 


Yes! 


Subject  D: 

Yes.  I  see  a  need  to  have  the  technical  team  provide  input  or  direction  on  the  cost.  Most 
contracting  officers  do  not  have  enough  knowledge  to  make  valid  decisions. 
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Appendix  D.  Value  Hierarchy  Definitions 


Price 

The  price  value  is  based  on  the  overall  factored  coefficient  value  submitted  by  the 
contractor.  A  coefficient  that  is  too  low  indicates  that  a  contractor  may  be  trying  to 
underbid  to  get  a  higher  evaluation,  but  may  not  be  successful  throughout  the  life  of  the 
contract.  An  extremely  high  coefficient  is  undesirable  because  the  Air  Force  does  not 
want  to  lose  money  throughout  the  life  of  the  contract  either.  The  intent  is  to  achieve  a 
reasonable  price  for  the  Government  while  at  the  same  time  ensuring  the  contractor  will 
be  successful. 

Past  Performance 

Past  performance  is  an  important  value  because  this  helps  the  Air  Force  determine 
the  amount  of  confidence  we  can  place  in  the  abilities  of  a  contractor  based  on  work 
performed  in  previous  (within  the  past  3  years)  contracts.  Evaluation  of  the  past 
performance  enables  the  Air  Force  to  determine  the  capabilities  of  a  contractor  in  tenns 
of  operations,  management,  and  construction  efforts  of  a  SABER  contractor.  The  RFP 
details  example  projects  that  a  contractor  can  expect  to  encounter,  therefore  there  should 
be  viable  example  of  similar  an  exceeding  efforts  submitted  by  the  contractors.  Past 
performance  is  further  broken  down  into  three  sub-values:  Competency,  Relevancy,  and 
Timely  Performance. 

Competency 

The  Air  Force  desires  a  SABER  contractor  that  has  demonstrated  the  ability  and 
competence  of  performing  and  managing  similar  construction  projects  to  those  that  will 
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be  encountered  throughout  the  lifespan  of  a  SABER  contract.  This  value  is  a  means  to 
measures  the  confidence  that  the  Air  Force  can  expect  to  place  on  a  contractor  based  off 
of  previous  performance  evaluations.  Competency  is  further  broken  down  into  two  sub- 
sub-values:  On-site  Capabilities  and  Customer  Satisfaction. 

On-site  Capabilities 

The  on-site  capabilities  of  the  competing  contractors  are  extremely  important  for 
a  SABER  contract  due  to  the  high  volume  and  quick  completion  of  construction  projects 
to  be  issued.  It  is  essential  that  the  Air  Force  has  the  ability  to  rate  the  capabilities  of  the 
contractors  to  ensure  that  the  expedition  factor  entailed  in  SABER  work  can  be 
accomplished  successfully. 

Customer  Satisfaction 

A  rather  generic  but  still  extremely  important  value  is  customer  satisfaction.  The 
Air  Force  seeks  contractors  that  will  not  only  do  the  work  required,  but  complete  the 
work  in  a  fashion  that  meet  or  exceeds  the  expectations  of  their  customers.  There  are 
numerous  Task  Orders  throughout  the  years  of  a  SABER  contract,  therefore  the 
contractor  need  to  demonstrate  a  fluid  and  consistent  high  standard  of  business  practice. 
The  Air  Force  is  an  organization  which  requires  a  standard  of  excellent,  and  will  accept 
nothing  less. 

Relevancy 

The  importance  and  utilization  of  SABER  as  means  of  construction  on  Air  Force 
bases  has  increased  by  leaps  and  bounds  over  the  years.  These  construction  projects  can 
and  will  vary  greatly  in  size,  scope,  and  manner  of  execution.  The  contractors  need  to  be 
rated  on  their  abilities  to  be  able  to  handle  large  scale,  multidiscipline  projects.  The  Air 
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Force  desires  a  well  rounded  contractor,  and  would  also  prefer  a  contractor  with  military 
installation  experience  due  to  the  unique  requirement  and  standards  that  are  entailed  with 
projects  performed  on  bases. 

Timely  Performance 

The  timeliness  of  perfonnance  which  can  be  expected  from  a  contractor  is  crucial 
for  a  SABER  contract.  The  main  purpose  of  SABER  is  to  plan  and  perform  construction 
projects  in  an  expedited  method.  The  contractors  need  to  present  the  ability  to  meet 
deadlines  and  handle  unforeseen  conditions  in  a  timely  and  efficient  manner.  The  intent 
of  having  a  SABER  contract  is  negated  by  contractors  that  do  not  show  the  ability  to 
perform  in  a  timely  fashion. 

Technical  Capabilities 

The  value  of  technical  capabilities  focuses  on  how  well  a  contractor  adheres  to 
regulations,  building  codes,  and  building  standards  as  well  as  managerial  issues.  Again, 
the  Air  Force  is  seeking  an  efficient,  ethical,  and  competent  contractor  who  can 
demonstrate  excellence  in  their  technical  capabilities.  The  value  of  technical  capabilities 
is  further  broken  down  into  four  sub-values:  Labor  Standards,  Safety  Standards, 
Management  Effectiveness,  and  Quality  Control. 

Labor  Standards 

This  value  is  incorporated  to  ensure  that  the  contractor  can  comply  with 
applicable  laws  and  regulations  governing  labor  standards.  The  Air  Force  will  not 
tolerate  non-compliance  of  such  issues  and  seeks  a  contractor  that  will  demonstrate  full 
adherence  to  applicable  labor  standards. 
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Safety  Standards 

Safety  is  a  major  concern  on  Air  Force  bases,  therefore  the  contractor  need  to 
have  the  ability  to  construct  and  implement  an  appropriate  safety  standard  for  their 
corporation.  This  value  enable  the  Air  Force  expressive  the  confidence  level  held  in  the 
contractor  to  be  able  to  run  operations  in  an  efficient  and  safe  application. 

Management  Effectiveness 

As  stated  throughout  the  definitions  of  previous  values,  a  SABER  contract  will 
require  a  maximum  managerial  effort  from  the  contractors  due  to  the  nature  and  intent  of 
the  SABER  program.  Several  projects  will  be  executed  simultaneously,  and  unforeseen 
site  conditions  and  change  orders  will  happen,  so  the  contractor  needs  to  demonstrate  an 
ability  to  handle  an  array  of  situations.  Additionally,  the  contractors  need  to  be  readily 
available,  or  have  a  representative  that  is  readily  available  to  address  issues  that  arise 
with  ongoing  projects  and  future  projects. 

Quality  Control 

The  Air  Force  desires  a  contractor  that  can  not  only  provide  timely  and  cost 
efficient  projects,  but  also  provide  coherent  plans  and  high  quality  work.  The  contractor 
needs  to  be  able  to  demonstrate  the  ability  to  perform  in  compliance  with  applicable 
regulations  and  building  codes,  as  well  as  an  ability  to  properly  submit  and  adhere  to 
submittal  requirements. 
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Appendix  E.  Measure  Definitions 


Ml:  Coefficient 


Category  Definition 

<1.15  or  1.35  < 

Overall  coefficient  submitted  by  contractor  is  less  than  1.15 
or  greater  than  1.35 

1.15-1.17 

Overall  coefficient  submitted  falls  in  corresponding  range 

1.18-1.20 

Overall  coefficient  submitted  falls  in  corresponding  range 

1.21  -  1.23 

Overall  coefficient  submitted  falls  in  corresponding  range 

1.24-1.26 

Overall  coefficient  submitted  falls  in  corresponding  range 

1.27-1.29 

Overall  coefficient  submitted  falls  in  corresponding  range 

1.30-1.32 

Overall  coefficient  submitted  falls  in  corresponding  range 

1.33-1.35 

Overall  coefficient  submitted  falls  in  corresponding  range 
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M2:  Primary  or  Subcontractor 


Category 

Category  Definition 

Neither 

Proposal  does  not  demonstrate  the  capacity  of  contractor  to 
effectively  conduct  and  manage  construction  work  as  a 
subcontractor  or  a  prime  contractor. 

Sub  Only 

Proposal  demonstrates  capacity  of  contractor  to  effectively 
conduct  and  manage  construction  work  as  a  subcontractor 
only. 

Both 

Proposal  demonstrates  capacity  of  contractor  to  effectively 
conduct  and  manage  construction  work  as  either  a 
subcontractor  or  a  prime  contractor. 

Prime  Only 

Proposal  demonstrates  the  capacity  of  contractor  to  effectively 
conduct  and  manage  construction  work  as  a  prime  contractor 
only. 
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M3:  In-house  Subcontract  out 


Category 

Category  Definition 

Neither 

Proposal  does  not  demonstrate  the  capacity  of  contractor  to 
effectively  conduct  construction  work  within  their  corporation 
nor  the  ability  to  effectively  acquire  and  manage 
subcontractor(s)  to  perform  construction  work. 

Subout  only 

Proposal  only  demonstrates  capacity  of  contractor  to 
effectively  acquire  and  manage  subcontractor(s)  to  perform 
construction  work. 

In-house  only 

Proposal  only  demonstrates  capacity  of  contractor  to 
effectively  conduct  construction  work  within  the  means  of 
their  corporation. 

Both 

Proposal  demonstrates  the  capacity  of  contractor  to  effectively 
perform  construction  work  either  by  the  means  of  their  own 
corporation  or  through  the  use  of  subcontractor(s). 
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M4:  Rehire 


Category 

Category  Definition 

No 

Proposal  does  not  demonstrate  the  desire  for  the  contractor  to 
be  rehired  for  future  work. 

Yes 

Proposal  demonstrates  the  dire  for  the  contractor  to  be  rehired 
for  future  work. 
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M5:  Cooperative 


Category 

Category  Definition 

Not 

Cooperative 

Proposal  demonstrates  negative  business  conduct  or  does  not 
address  cooperative  capabilities  at  all. 

Satisfactory 

Proposal  addresses  cooperative  capabilities  but  demonstrates 
inconsistencies  of  both  negative  and  positive  conduct. 

Good 

Proposal  addresses  cooperative  capabilities  in  strictly  positive 
manner. 

Exceptional 

Proposal  addresses  outstanding  positive  cooperative 
capabilities  to  include  incentives  exercised  to  maintain  or 
improve  capabilities. 
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M6:  Contracting  Actions  Against 


Category 

Category  Definition 

0-1 

Proposal  demonstrates  contracting  actions  against  the 
contractor  to  number  in  the  corresponding  range. 

2-4 

Proposal  demonstrates  contracting  actions  against  the 
contractor  to  number  in  the  corresponding  range. 

5-6 

Proposal  demonstrates  contracting  actions  against  the 
contractor  to  number  in  the  corresponding  range. 

More  than  6 

Proposal  demonstrates  contracting  actions  against  the 
contractor  to  be  greater  than  6. 
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M7:  Scope  Multidiscipline 


Category 

Category  Definition 

<3 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  less  than  three  construction  disciplines. 

3 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  three  construction  disciplines. 

4-5 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  four  or  five  construction  disciplines. 

5  < 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  five  or  more  construction  disciplines. 
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M8:  Size  $  Amount 


Category 

Category  Definition 

<  100K 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  construction  projects  priced  at  less  than  $100K  only. 

100K-250K 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  construction  projects  priced  between  $100K  and 
$250K. 

251K-500K 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  construction  projects  priced  between  $25  IK  and 
$500K. 

500K  < 

Proposal  demonstrates  the  contractor’s  ability  to  properly 
execute  construction  projects  priced  in  excess  of  $500K. 
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M9:  Military  Instillation  Experience 


Category 

Category  Definition 

No 

Proposal  does  not  demonstrate  contractor’s 
familiarity/previous  work  experience  with/on  military 
installations  and  conditions  therein. 

Yes 

Proposal  demonstrates  contractor’s  familiarity/previous  work 
experience  with/on  military  installations  and  conditions 
therein. 
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M10:  Met  Performance  Standards 


Category 

Category  Definition 

Not  Met 

Proposal  demonstrates  negative  ability  of  contractor  to  meet 
performance  standards  or  does  not  address  ability  to  meet 
performance  standards  at  all. 

Satisfactory 

Proposal  demonstrates  inconsistencies  in  ability  of  contractor 
to  meet  performance  standards,  but  relays  overall  positive 
ability  of  contractor  to  meet  performance  standards. 

Good 

Proposal  demonstrates  ability  of  contractor  to  meet 
performance  standards  in  a  strictly  positive  manner. 

Exceptional 

Proposal  demonstrates  contractor  ability  to  achieve 
performance  standards  in  a  manner  which  exceeds  the 
expectations  of  the  end  users/customers. 
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Mil:  Submittals  in  Timely  Manner 


Category 

Category  Definition 

On  Time 

Proposal  demonstrates  the  ability  of  the  contractor  to 
consistently  delivery  submittals  on  time. 

1  -  7  Days 
Late 

Proposal  demonstrates  the  contractor  to  consistently  deliver 
submittals  1  to  7  days  after  deadline. 

8-14  Days 
Late 

Proposal  demonstrates  the  contractor  to  consistently  deliver 
submittals  8  to  14  days  after  deadline. 

14  Days  Late  < 

Proposal  demonstrates  the  contractor  to  consistently  deliver 
submittals  more  than  14  days  after  deadline. 
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M12:  Resolve  Delays 


Category 

Category  Definition 

None 

No  performance  record  to  resolve  delays  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  resolve  delays  to  the  required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  resolve  delays  to  the  required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  resolve  delays  to  the  required 
effort. 
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M13:  Completion  of  Punch-list  Items 


Category 

Category  Definition 

None 

No  performance  record  to  complete  punch-list  items 
identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  complete  punch-list  items  to  the  required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  complete  punch-list  items  to  the  required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  complete  punch-list  items  to  the 
required  effort. 
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M14:  Warranty  in  Timely  Manner 


Category 

Category  Definition 

On  Time 

Proposal  demonstrates  the  ability  of  the  contractor  to 
consistently  address  warranty  issues  to  the  complete 
satisfaction  of  customer  expectations. 

1  -  7  Days 
Late 

Proposal  demonstrates  the  contractor  to  consistently  address 
warranty  issues  1  to  7  days  late  of  customer  expectations. 

8-14  Days 
Late 

Proposal  demonstrates  the  contractor  to  consistently  address 
warranty  issues  8  to  14  days  late  of  customer  expectations. 

14  Days  Late  < 

Proposal  demonstrates  the  contractor  to  consistently  address 
warranty  issues  more  than  14  days  late  of  customer 
expectations. 

116 


M15:  Compliance  Laws  &  Regs 


Category 

Category  Definition 

None 

No  performance  record  of  compliance  with  applicable  labor 
laws  and  regulations  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  comply  with  applicable  labor  laws  and 
regulations  to  the  required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  comply  with  applicable  labor  laws  and 
regulations  to  the  required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  comply  with  applicable  labor  laws 
and  regulations  to  the  required  effort. 
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M16:  Safety  Plan 


Category 

Category  Definition 

None 

No  performance  record  to  complete  and  properly  execute  a 
safety  plan  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  complete  and  execute  a  safety  plan  to  the 
required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  complete  and  execute  a  safety  plan  to  the 
required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  complete  and  execute  a  safety  plan 
to  the  required  effort. 
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M17:  Contract  Requirements 


Category 

Category  Definition 

None 

No  performance  record  to  complete  contract  requirements 
identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  complete  contract  requirements  to  the  required 
effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  complete  contract  requirements  to  the  required 
effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  complete  contract  requirements  to 
the  required  effort. 
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M18:  Ability  to  Reduce  Problems 


Category 

Category  Definition 

None 

No  performance  record  showing  ability  to  reduce  and  remedy 
problems  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  reduce  and  remedy  problems  to  the  required 
effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  reduce  and  remedy  problems  to  the  required 
effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  reduce  and  remedy  problems  to 
the  required  effort. 
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M19:  On-site  Presence 


Category 

Category  Definition 

None 

No  performance  record  to  execute  and  maintain  on-site 
presence  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  execute  and  maintain  on-site  presence  to  the 
required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  execute  and  maintain  on-site  presence  to  the 
required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  execute  and  maintain  on-site 
presence  to  the  required  effort. 
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M20:  Quality  Control  Plan 


Category 

Category  Definition 

None 

No  performance  record  to  complete  and  properly  execute  a 
quality  control  plan  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  complete  and  execute  a  quality  control  plan  to 
the  required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  complete  and  execute  a  quality  control  plan  to 
the  required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  complete  and  execute  a  quality 
control  plan  to  the  required  effort. 
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M21:  Quality  Workmanship 


Category 

Category  Definition 

None 

No  performance  record  demonstrating  the  ability  of  contractor 
to  perfonn  quality  workmanship  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  perform  quality  workmanship  to  the  required 
effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  perform  quality  workmanship  to  the  required 
effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  perform  quality  workmanship  to 
the  required  effort. 
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M22:  Materials  Listed  Specified 


Category 

Category  Definition 

None 

No  performance  record  of  adherence  to  listed/specified 
materials  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  adhere  to  listed/specified  materials  to  the 
required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  adhere  to  listed/specified  materials  to  the 
required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  adhere  to  listed/specified  materials 
to  the  required  effort. 
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M23:  Adequate  Submittals 


Category 

Category  Definition 

None 

No  performance  record  of  providing  adequate  submittals 
identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  provide  adequate  submittals  to  the  required 
effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  provide  adequate  submittals  to  the  required 
effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  provide  adequate  submittals  to  the 
required  effort. 
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M24:  Compliance  Regs  and  Code 


Category 

Category  Definition 

None 

No  performance  record  of  compliance  with  applicable  building 
regulations  and  code  identifiable. 

Satisfactory 

Proposal  demonstrates  that  some  doubt  exists  that  the 
contractor  will  comply  with  applicable  building  regulations 
and  code  to  the  required  effort. 

Good 

Proposal  demonstrates  that  little  doubt  exists  that  the 
contractor  will  comply  with  applicable  building  regulations 
and  code  to  the  required  effort. 

Exceptional 

Proposal  demonstrates  that  essentially  no  doubt  exists  that  the 
contractor  will  successfully  comply  with  applicable  building 
regulations  and  code  to  the  required  effort. 
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Appendix  F.  Alternative  Scores 


Alternative 

Ml 

M2 

M3 

M4 

M5 

Contractor  A 

1.21  -  1.23 

Both 

Yes 

Good 

Contractor  B 

1.18-1.20 

Both 

Yes 

Satisfactory 

Contractor  C 

<  1.15  or  1.35  < 

Prime  Only 

Yes 

Good 

Contractor  D 

1.15-1.17 

Both 

Yes 

Satisfactory 

Contractor  E 

1.21  -  1.23 

Both 

Yes 

Exceptional 

Contractor  F 

1.24-1.26 

Prime  Only 

Yes 

Good 

Contractor  G 

1.27-1.29 

Prime  Only 

Yes 

Good 

Alternative 

M6 

M7 

M8 

M9 

M10 

Contractor  A 

5  < 

500K  < 

Satisfactory 

Contractor  B 

5  < 

500K  < 

Yes 

Satisfactory 

Contractor  C 

2-4 

4-5 

500K  < 

Good 

Contractor  D 

0-1 

5  < 

500K  < 

Yes 

Satisfactory 

Contractor  E 

0-1 

5  < 

500K  < 

Yes 

Good 

Contractor  F 

2-4 

4-5 

500K  < 

Yes 

Satisfactory 

Contractor  G 

0-1 

5  < 

100K-250K 

Yes 

Good 

Alternative 

Mil 

M12 

M13 

M14 

Contractor  A 

On  Time 

Exceptional 

Exceptional 

On  Time 

Contractor  B 

8-14  Days  Late 

Satisfactory 

Satisfactory 

On  Time 

Contractor  C 

On  Time 

Good 

Good 

On  Time 

Contractor  D 

On  Time 

Satisfactory 

Satisfactory 

1  -  7  Days  Late 

Contractor  E 

On  Time 

Exceptional 

Exceptional 

On  Time 

Contractor  F 

1  -  7  Days  Late 

Good 

Good 

8-14  Days  Late 

Contractor  G 

On  Time 

Satisfactory 

Good 

On  Time 
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Alternative 

M15 

M16 

M17 

M18 

M19 

Contractor  A 

Good 

Exceptional 

Good 

Exceptional 

Exceptional 

Contractor  B 

Satisfactory 

Satisfactory 

Satisfactory 

Satisfactory 

Satisfactory 

Contractor  C 

Good 

Good 

Good 

Exceptional 

Exceptional 

Contractor  D 

Satisfactory 

Satisfactory 

Satisfactory 

Satisfactory 

Good 

Contractor  E 

Good 

Exceptional 

Exceptional 

Good 

Exceptional 

Contractor  F 

Exceptional 

Good 

Good 

Good 

Good 

Contractor  G 

Satisfactory 

Satisfactory 

Good 

Satisfactory 

Good 

Alternative 

M20 

M21 

M22 

M23 

M24 

Contractor  A 

Exceptional 

Exceptional 

Exceptional 

Good 

Good 

Contractor  B 

Satisfactory 

Good 

Good 

Good 

Good 

Contractor  C 

Good 

Good 

Good 

Exceptional 

Exceptional 

Contractor  D 

Good 

Satisfactory 

Satisfactory 

Satisfactory 

Satisfactory 

Contractor  E 

Good 

Exceptional 

Exceptional 

Exceptional 

Exceptional 

Contractor  F 

Good 

Good 

Good 

Good 

Good 

Contractor  G 

Satisfactory 

Satisfactory 

Good 

Good 

Satisfactory 
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Appendix  G.  Final  Questionnaire 


Question  1 .  How  does  the  current  evaluation  process  of  SABER  Source  Selection 
compare  to  the  VFT  model? 


Answers: 

Subject  A: 

[Current  evaluation  process]  Much  more  subjective,  tends  to  lock  you  into  low  price 
offerors  and  puts  tremendous  pressure  on  the  selection  team  to  definitively  prove  that  the 
low  price  guy  isn’t  the  best  or  else  you’re  stuck  with  him.  The  VFT  model  helps  make  a 
strong  case  for  why  the  selected  offeror  is  best  choice  and  this  will  also  help  if  there  is 
ever  a  protest. 


Subject  B: 

The  current  process  focuses  on  price  and  performance.  There  is  no  regard  for  staff  size 
or  professional  expertise. 


Subject  C: 

The  VFT  model  makes  it  much  easier  to  select  factor  to  be  used  in  the  selection  process 
and  to  understand  their  value  as  related  to  the  total  value.  The  entire  process  is  much 
easier  to  evaluate  contractors. 


Subject  D: 

The  current  evaluation  process  is  very  subjective  and  does  not  provide  a  very  user 
friendly  result.  Explaining  the  rational  of  your  decisions  is  very  difficult  without  some 
type  of  quantifying  number.  VFT  is  very  detailed  and  does  provide  the  desired 
quantifying  result. 
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Question  2:  Is  there  a  clear  understanding  of  all  evaluation  criteria  utilized  in  the  VFT 
model? 


Answers: 

Subject  A: 

Yes. 

Subject  B: 

Yes.  The  criteria  and  evaluation  can  be  used  through  empirical  data. 

Subject  C: 

It  is  easier  to  understand  the  values  of  the  factors.  The  bar  graph  makes  it  very  clear  to 
understand  the  values  and  their  relationships. 

Subject  D: 

Yes.  It  provides  very  descriptive  breakouts  of  each  factor  and  generates  thought. 
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Question  3:  How  many  times  would  competing  contractors  need  to  be  evaluated  or 
re-evaluated  utilizing  the  VFT  model? 


Answers: 

Subject  A: 

Unsure. 

Subject  B: 

(1)  One  time. 

Subject  C: 

One  time!  Very  good. 

Subject  D: 

I  do  not  see  much  need  to  do  this  if  the  results  are  so  descriptive.  No  more  than  once. 
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Question  4:  Is  there  enough  information  provided  by  the  VFT  model  to  conduct  proper 
documentation? 


Answers: 

Subject  A: 

Yes. . .  much  better  than  current  process. 

Subject  B: 

Yes. 

Subject  C: 

The  ability  to  select  additional  factors  and  assign  value  to  these  and  how  each  relates  to 
the  total  value  and  to  the  other  proposals  is  very  good  documentation. 

Subject  D: 

Yes. 
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Question  5:  What  information,  if  any,  needs  to  be  more  evident  when  utilizing  the  VFT 
model? 


Answers: 

Subject  A: 

Nothing  that  I  can  think  of. 

Subject  B: 

1 .  Capability  of  Contractors 

2.  Expertise  of  Contractors 

3.  Disciplines  involved 

Subject  C: 

I  was  not  able  to  identify  further  information  needs  at  this  time. 

Subject  D: 

None. 
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Question  6:  Is  a  VFT  model  a  viable  tool  for  the  evaluation  process  of  SABER  Source 
Selection? 


Answers: 
Subject  A: 


Yes. 


Subject  B: 

Yes.  Much  better  than  our  current  evaluation  system. 


Subject  C: 

This  would  be  a  very  useful  tool  to  use  in  SABER  Source  Selections.  I  feel  this  model 
would  greatly  improve  the  selection  process  and  documentation  needs  to  support 
contractor  selection. 


Subject  D: 

Yes,  I  would  utilize  this  tool  now  if  it  was  made  available. 
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